17 ms·

Go: Redefining For Loop Variable Semantics

464 points by denysvitali 4y ago

socialdemocrat 4y ago
Yeah… I seem to remember scratching my head because of a variation of this problem. It was kind of driving me nuts. Yet the number of moments like this has been low for me in Go compared to C++.
jupp0r 4y ago
How does this work, in this code: var all []\*Item for _, item := range items { all = append(all, &item) } When &item is the same for all iterations, that means that it's pointing to the same memory address. Is each item in items copied to this address prior to each iteration body invocation? This seems strange as this copy could potentially be very expensive. What am I missing?
- silisili 4y ago
  Nothing, you got it. Most of the time you're iterating a slice of pointers, though, so only the address gets copied. And in those cases, this bug doesn't exist(unless of course you're going from * to ** for some reason).
- foldr 4y ago
  To add to the sibling, you could iterate by index to avoid the copy: for i := range items { ptr := &items[i] ... } In this case of course you also get different semantics. The ptr variable is bound to the address of each of the original Item values in the slice, whereas in the code in your comment, &item is the address of a single heap-allocated Item variable.
- ytch 4y ago
  Could we assume it is like executing: var item Item item = items[0] all = append(all, &item) item = items[1] all = append(all, &item) ...
- skywhopper 4y ago
  You aren’t missing anything. How expensive the copy is depends on the type of item. This is why you should probably use pointers in slices you’re going to loop over instead of the object itself.
zebracanevra 4y ago
If you're at all interested in how for loops and scope work in Javascript, Jake Archibald and Surma have a great video on the topic: https://www.youtube.com/watch?v=Nzokr6Boeaw https://www.youtube.com/watch?v=Nzokr6Boeaw Similar changes were made in newer versions of ES so that the for loop in this article works out of the box, like C#. Slightly off-topic to this article: I wish "do while" loops had the "while" condition in the inner scope, not the outer scope. So many times I have wished that I could access the inner scope... I end up using a while(true) with an if { break; } at the end instead in 99% of cases where a do while could've been the perfect thing...
- chrismorgan 4y ago
  Except the ECMAScript changes weren’t changes but additions. Anything using the var keyword gets the traditional semantics, anything using let and const gets lexical scoping. As an example: x = []; for (var i of [1, 2, 3]) { x.push(() => i); } x.map(f => f()) == [3, 3, 3] x = []; for (let i of [1, 2, 3]) { x.push(() => i); } x.map(f => f()) == [1, 2, 3] What’s being proposed for Go is instead a breaking change.
  - baq 4y ago
    > What’s being proposed for Go is instead a breaking change. given the context, it might actually be a fixing change...
  - foldr 4y ago
    >What’s being proposed for Go is instead a breaking change. It sounds like you only get the breaking change in modules that require above a certain version of Go. So this should not break old code. It's perhaps more analogous to the way that "use strict" in JavaScript 'breaks' parsing of octal constants.
    - chrismorgan 4y ago
      Ah, my apologies; I missed that part. (I last used Go a decade ago, long before go.mod.) Then this seems fairly reasonable, given also how rarely things will depend on one way or the other.
hardware2win 4y ago
How did they even end up with such a crazy behaviour?
- mpartel 4y ago
  Several languages have or have had this behaviour. C# made a similar change to its 'foreach' loops a long time ago (but not 'for' loops for some reason). I suppose it's the natural way to do it when implementing a language and not thinking about it too much. It makes a for-loop equivalent to a simple while loop with the loop vvariable initialized outside of the loop.
  - hardware2win 4y ago
    In c# you mean closure capture?
    - masklinn 4y ago
      Almost certainly, that is the vehicle for the issue in most languages.
  - Blikkentrekker 4y ago
    To be fair it is no more difficult to implement it with the variable initialized inside of it. In fact, it's easier to.
  - int_19h 4y ago
    The C# change is described in detail in the linked GitHub issue - one of C# devs left a comment explaining it. TL;DR is that both "for" and "foreach" scoping fixes would be breaking changes, but "foreach" was easier to justify because it was already a C#-specific construct syntactically, unlike "for" which uses the same exact syntax as C, Java etc, and they were very sensitive to backwards compatibility at the time (esp. since the tooling didn't have the ability to target various language versions within the same project easily). At the same time, "foreach" represented the vast majority of breakage when they looked at existing code, perhaps because the scoping in classic "for" is more obvious due to the fact that variable mutation is explicit there.
jstimpfle 4y ago
WTF. I haven't been interested in Go after looking at it briefly a decade ago. This post brings back the weird feeling about it that I had then. It looks like taking the address of a local variable (item := item; append(..., &item);) and using that outside the scope of the local variable. But apparently that is ok. Variations of such perceived weirdness exist in many other languages with complicated "object models" as well to be fair. Delphi has some strange adressing stuff going on as well. Python has this weird "default list" thing. Most object languages don't let you take the address of something (like the Go example shows) at all, but have only object references which I find unergonomic.
- silon42 4y ago
  I also feel there is a problem in that line, not necessarily just a for loop. Maybe that's my C/C++/Rust experience.
  - tialaramex 4y ago
    In Rust we know we only gave the append function an immutable reference which lives until the next iteration. If append is OK with that, we're golden. Maybe internally it clones these references -shrug. If append() needs the thing, not just an immutable reference which expires soon, its signature would demand we move one into it, and we don't have one so we'd need to e.g. make one with Clone.
    - maccard 4y ago
      > Maybe internally it clones these references -shrug. That's what go does under the hood in 95% of cases. If I do: func foo(a int) *int { return &a } func main() { fmt.Println(*foo(3)) } It does the right thing. It's just that in this case (and in lambdas) they got it wrong.
    - masklinn 4y ago
      TBF Rust also intentionally used the semantics Go is migrating to from the start, because: - experience with the issue in languages with wider scoping e.g. it’s a common issue in JS, as well as Python (though slightly less so) - Rust’s iterators were originally internal so that was pretty natural (1) is also why for(let and for(const have different scoping than for(var in JS: `var` has function scoping, `let` and `const` were introduced with block scoping, and for loops they were specifically specced with “inner” (per-iteration) scope.
fomine3 4y ago
It's great that who worked for similar changes on C#, responses immediately.
corsix 4y ago
Lua gets this right - the lowering of loops (e.g. https://www.lua.org/manual/5.1/manual.html#2.4.5 https://www.lua.org/manual/5.1/manual.html#2.4.5) says “var is invisible” and has “local v = var”, the latter akin to Go’s “item := item”
benhoyt 4y ago
I'm an experienced Go programmer and still accidentally do this every so often. It certainly bit me and my team several times when we were newish to Go. And particularly for newbies, it's hard to debug and understand when you do run into it. All of which to say I'm really glad they're trying to fix this.
- tezza 4y ago
  Yep, bitten by this one on repeat myself. definitely a WTF moment each time.
  - kubb 4y ago
    same here, happened several times when launching a goroutine inside the loop the fix was usually adding x := x before go func() { do something with x }
    - masklinn 4y ago
      A common alternative fix is to use IIFE, since the expression must be a call. `go` evaluates everything but the final function call in the context of the caller. So go func (x int) { … } (x) Will do the same, and is easier to extract to a named function.
      - Joker_vD 4y ago
        The latter is also easier to re-inline accidentally because go func (x int) { do_work(x) } (x) seems like a very roundabout way to say go do_work(x) The "x := x" solution too suffers from this problem but slightly less: both idioms look like they are no-ops (while they are actually not) but at least "x := x" is weird enough to look like it was a deliberate choice, not some vestige from refactoring.
        masklinn 4y ago
        Er… it makes no difference? Your first version is in fact a worse way to write the second one.
zendist 4y ago
Thrilled to see Jared Parsons of the C# team pitch in and provide some perspective on how things were done for C#5 when a similar change was made. Kudos Jared!
- pjmlp 4y ago
  Same here, DevDiv is now polyglot focused, so you will see regular comments from .NET folks on other languages as well (mainly Java and Go). David Fowler tends to tweet every now and then about them as well.
- Cthulhu_ 4y ago
  Yeah, it's often a joy to read these posts and the perspectives on it; this is how programming language development should be done.
- int_19h 4y ago
  What's interesting is that C# 5 release (which made the breaking change) was back in 2012, and both the change and the reasons for it were very widely discussed at the time. This is right around the time when Go shipped its 1.0, and it's kinda surprising that they either didn't look closely at "near-peer" languages, or if they did, couldn't see how this problem was fully applicable to their PL design, as well. (Note that C# at least had the excuse of not having closures in the first version, which makes scoping of "foreach" moot - the problem only showed up in C# 2.0. But Go had lambdas from the get-go, so this interaction between loops and closures was always there.)
  - pjmlp 4y ago
    Well, this isn't the only language feature where Go designers ignored the path trailed by other languages. Still looking forward to the day they will discover Pascal enumerations.
colonwqbang 4y ago
item := item What does this do? That should be a no-op by default in any sane language.
- _ikke_ 4y ago
  The := both declares a new variable by an inferred type and assigns a value to it. Explicitly, this would be the same as var item *Item = item It creates a locally scoped variable that shadows the existing one.
  - colonwqbang 4y ago
    Ah ok, so different name resolution rules apply on the left and right sides of the operator? On the left side, "item" resolves to the "item" in inner scope (that is declared on the same line). On the right side it resolves to the "item" in outer scope. Then I understand what's happening though I'm still not convinced it's sane behaviour. I would expect all identical names in a single scope to refer to the same variable.
    - deleted 4y ago
      [deleted]
- ithkuil 4y ago
  item = item That is a no-op a := b Declares a new variable a and assigns it. It's sugar for: var a = b So read it as: var item = item Many languages have scopes and variable shadowing. You can argue this is bad practice or whatever, but it's only confusing if you think := is assignment which is covered in the first page of the Go language tutorial.
  - valenterry 4y ago
    Not a go dev, but var item = item should still be a noop imo. If it does some magic like dereferencing, make it explicit. And no, other languages making the same mistake is not a good excuse. Especially not for a language focused on inexperienced newgrads.
the_gipsy 4y ago
> Loop variables being per-loop instead of per-iteration is the only design decision I know of in Go that makes programs incorrect more often than it makes them correct. Since it is the only such design decision, I do not see any plausible candidates for additional exceptions. The bar is quite low.
- dontlaugh 4y ago
  It's also very subjective. Just off the top of my head I can think of: - zero values instead of sum types - nil, especially on interfaces - goroutines sharing memory - goroutine panics not being raised in the spawner by default
- seer-zig 4y ago
  golang has quite the suspect decisions that show up when writing large programs https://www.uber.com/blog/data-race-patterns-in-go/ https://www.uber.com/blog/data-race-patterns-in-go/
  - cesarb 4y ago
    Working link: https://www.uber.com/en-US/blog/data-race-patterns-in-go/ https://www.uber.com/en-US/blog/data-race-patterns-in-go/ (For me, the link you posted does a 302 redirect to https://www.uber.com/pt-BR/blog/data-race-patterns-in-go/ https://www.uber.com/pt-BR/blog/data-race-patterns-in-go/ which gives me a 404 error page. It's a bit insane that whether the link you posted works or not depends on your locale, and unfortunately this is not the first time I've seen this kind of baffling redirect misbehavior.)
hknmtt 4y ago
Strange, never had a problem with this. If I need to reference non-pointer I just use key(append(foo, items[k])) but i find it weird that the print example is not working since in loop the value is passed by copy, just like everywhere else in go, so it should work correctly. as for pointer, again, pass by copy. I see no problem here, it either never bit me in my ass or i figured it out soon in my Go journey.
bjoli 4y ago
I have said this before, but a source->source compiler is a very handy thing to have. A loop construct in chez or guile scheme would probably translate into something tail recursive. You could then macro-expand and optimize the loop to the code that is then compiled into lower level. I have been in situations where I have had to expand a macro to figure out what is going on. Not having it in a situation like this (where a for loop is obviously just a goto or a tail call) is usuallly a pain in the ass. If it was translated into the same language it would also be easier to define what it should translate to.
- valenterry 4y ago
  I'd rather use a language from the beginning that isn't made up accommodate tons of newgrads as fast as possible on a big codebase. That is what Google needs, but most people are better of with more advanced languages. But people follow Google just like they did with Angular. I predict Go will follow the same destiny, will just take longer. The tech debt is still piling up.
shp0ngle 4y ago
If you never did a mistake connected to this and caused a bug in real prod code, you haven’t been writing go for long enough.
- barsonme 4y ago
  Ehhhh, I’ve been writing Go for ~10 years and I’ve never had this make it into prod. Many times in local development and testing, though.
- Traubenfuchs 4y ago
  Ideally at least one of the tests out of your testing pyramid would catch something like that though.
  - Cthulhu_ 4y ago
    Ideally yeah, but unfortunately very few people work in ideal circumstances.
- bandrami 4y ago
  I internalized early on that the likelihood of this mistake means Go wants me to refactor when I find myself liable to making it.
- ainar-g 4y ago
  Or you've been using the looppointer analyser[1] from the start, heh. [1]: https://github.com/kyoh86/looppointer https://github.com/kyoh86/looppointer
- viiralvx 4y ago
  Yeahhhhh, I worked at a big company on their CI product and I had this bug bite me and break builds for a little bit. Luckily caught my mistake and fixed it within 15 minutes, but it's always easy to overlook.
bborud 4y ago
That was a very well written issue.
- mikepurvis 4y ago
  Shockingly high quality discussion too— have the comment moderation tools on Github gotten better in recent years, or is this just the golang community being awesome?
xmprt 4y ago
I'm all for this change. I've had to explain this weird programming quirk of Go to coworkers on multiple occasions and can't wait for it to be solved.
- Cthulhu_ 4y ago
  I'm just happy that it's one of the few weird quirks of Go. In my spheres, Javascript is the most commonly used language, and the 'wat' presentation is frequently referenced to.
mftb 4y ago
I'm happy the Go team takes breaking changes seriously. They should go ahead and fix this. The current semantics are lame.
anotherhacker0 4y ago
Anyone know the status of overall golang 2? also anyone know of any proposals around generics or pattern matching?
- mseepgood 4y ago
  There are no plans for a literal version 2 anymore. Generics were released in March. There are no active proposals for pattern matching.
  - gus_leonel 4y ago
    A quick guide to the generics support that's been added to Go: https://bitfieldconsulting.com/golang/generics https://bitfieldconsulting.com/golang/generics
  - jerf 4y ago
    Specifically, Golang 2.0 was reserved for backwards incompatible changes, generally understood to be of some significant size, not twiddly bits like this that are adequately covered by a go.mod flag. Generics was thought to possibly involve backwards incompatible changes, but it was done without backwards incompatibility. There aren't any similarly-sized issues on the horizon right now that would result in a 2.0. If you track the issues closely on GitHub, there's a constant low-level discussion going on about it, but so far nothing seems to be both important enough and large enough to justify the version change, like, not even close.
tgv 4y ago
I'm absolutely not looking forward to potentially breaking changes. Stability is such an important selling point (*). This is not something that can be fixed by a search-replace or a refactoring operation. And it's just begging for problems in the community. (*) please don't reply with "then you should keep the version below 1.20"
- mananaysiempre 4y ago
  Given a tool sophisticated enough to track variable scope (not a tall order), this absolutely can be fixed mechanically.
- masklinn 4y ago
  > please don't reply with "then you should keep the version below 1.20" That is literally the solution. Just remove your go.mod or keep it under 1.20, and you’ll keep the old behaviour forever. It’s the same as e.g. Rust’s edition, if you hate your life you can keep using Rust 2015 semantics to this day, and forever.
  - tgv 4y ago
    I knew there would be someone. So if you want a feature in a higher version, update your code base. Yeah, that's what stability doesn't mean. And "it's your problem now" isn't great Go advocacy either.
    - masklinn 4y ago
      There is no feature, it only sets the static evaluation of the language itself. Think for a second, if the go.mod stricture locked in the entire thing, you’d just be told to not upgrade. The entire point of the system is that it’s possible to update language-level semantics while allowing for the rest to progress for everybody. Setting go.mod means you get stdlib updates, but the semantics of the for loop does not change. And if the designers decide to add similar breaking changes in the future (change string literals or whatever), you’ll also be ignoring those.
crnkofe 4y ago
I gave this post all the upvotes and hearts I got. It's one of those weird Go language quirks where developer expectations just never match what's going on behind the scene. I still often find myself always looking at the loops if they actually modify/use correct variables.
zadwang 4y ago
If I understand correctly the fix requires the new code to add a line in go.mod to use the new behavior. This is about the same as adding x:=x in the loop, and more hidden. Not good. The alternative can be that whenever address of iteration variable is used inside the loop the variable is per iteration and otherwise it is per loop. This way it is not breaking the old code and have new semantics.
- zimpenfish 4y ago
  > whenever address of iteration variable is used That wouldn't fix the closure problem where the address isn't taken.
  - zadwang 4y ago
    You are right. Would it be possible to add closure use as another case?
    - zimpenfish 4y ago
      I guess you could but now you have to special-cases that need static analysis and all that jazz. Probably easier to just put it behind a version flag.
- viraptor 4y ago
  > This is about the same as adding x:=x in the loop It's one per module though. In large enough modules you'll have tens of x:=x though. I assume this also opens the option of doing more such changes through the same system in the future.
  - masklinn 4y ago
    And assuming new projects get the new mod by default, it also makes the langage default-safe, as opposed to requiring the cognitive overhead of evaluating individual loops, or requiring this nonsense as explicit prolog to every loop.
  - 4y ago
tialaramex 4y ago
> we gave the general rule that language redefinitions like what I just described are not permitted Not just a "general rule" that document also specifically talks about precisely this issue (for loops) and resolves that Go will not fix this. Change is the only constant, we should design systems with the expectation that they'll need to adapt over time or they will be replaced by something which can. With this mindset, Go should have solved the for loop problem years ago, just as C# did. This could have been a story about how once upon a time Go had these very silly for loop semantics, but that hasn't been true for many years. This necessity of change is why I think the decision not to take Epochs for C++ 20 was much more consequential than things like rejecting the "Goals and priorities" paper which had immediate effects (in that particular case spurring the Carbon experiment).
- morelisp 4y ago
  Arguably if Go had done this years ago, go.mod would not have existed and this would have an even bigger impact. Just like we'll rarely pick exactly the right behavior, we'll rarely pick exactly the right time to fix the broken behavior.
  - varispeed 4y ago
    How one depends on another? To me it sounds these things are orthogonal.
    - masklinn 4y ago
      Because of time? How would you use go.mod when go mod does not exist?
    - room271 4y ago
      go.mod provides a way to introduce the new behaviour incrementally. The key point is the dependencies can (and already do) declare different go versions in their go.mod file. When everything is compiled, each module is compiled with the behaviour that applies to its own declared go version. So even if you update your go.mod go version to take advantage of this (soon to be) new behaviour, you can continue to use existing deps happily without worrying they will break.
BoppreH 4y ago
The same problem frequently bites me in Python too: functions = [] for i in range(10): functions.append(lambda: print(f'Hello {i}')) for fn in functions: fn() # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 I find it one of Python's biggest warts because it's silent, hard to troubleshoot (especially the first time!), and like in Go the most straightforward fix looks like a mistake (i=i): for i in range(10): functions.append(lambda i=i: print(f'Hello {i}')) I guess this is a lesson in designing language semantics that match people's intuitions, and learning from previous languages' mistakes.
- Simplicitas 4y ago
  Yup. "Match [human] intuition" is the operative phrase in all of this. Thanks. Hope Go addresses this soon.
- Semaphor 4y ago
  If I do the same with for instead of foreach in C#, my IDE gives me a warning: "Captured variable is modified in the outer scope". Isn’t that the case for Python?
  - ketzu 4y ago
    Copy pasting OPs code to pycharm: No warning.
    - Semaphor 4y ago
      Probably worthy of a YouTrack report ;)
    - jstx1 4y ago
      Pylint catches it - https://pylint.pycqa.org/en/latest/user_guide/messages/warning/cell-var-from-loop.html https://pylint.pycqa.org/en/latest/user_guide/messages/warni...
  - maxloh 4y ago
foldl2022 4y ago
Glad that Nim had solved this problem with a macro `capture`. It's up to the developers to capture loop variables or not.
brudgers 4y ago
Oh, schadenfreude of Turing's Tarpit I giggle a bit http://www.lispworks.com/documentation/lw51/CLHS/Body/m_loop.htm http://www.lispworks.com/documentation/lw51/CLHS/Body/m_loop...
- assbuttbuttass 4y ago
  Lisp suffers from this same defect, maybe that was your point? What does the following code print out? (defvar refs (loop for i from 1 to 10 collect (lambda () (format t "~d~%" i)))) (loop for f in refs do (funcall f))
thiht 4y ago
I fail to see any downside to changing this semantics. This has honestly always felt like a language design bug more than anything. Having to write foo := foo at the beginning of a loop for it to behave as expected is a strong design smell. The gradual breaking (of fix depending on your point of view) with explicit opt-in looks great to me.
esjeon 4y ago
Am I the only one who thought about currying here? I mean, neither the ticket nor the thread mentions it. I don't think it's such an alien concept to people. TBH, with my full sincerity, this isn't really a way to solve the problem. Currying has been used for decades for reasons. For-loop isn't the only place where you introduce outer-scope variables into an immediate function. You can always have other variables involved, and you'll still have to be careful. Even though this change will reduce the absolute number of bugs, people will still have the same issue.
- justinpombrio 4y ago
  Currying is the observation that a function that takes a pair as an argument, `(a, b) -> c`, is equivalent to a function that returns a function: `a -> (b -> c)`. What's that have to do with for loops?
- quietbritishjim 4y ago
  Are you sure you mean currying? Currying is where you transform a function taking multiple arguments into a function that only takes one of the arguments and returns another function which takes one of the remaining arguments which (...recursively). Maybe you meant partial function application? Even that doesn't seem relevant, given that the function is actually being called immediately in the first example.
  - esjeon 4y ago
    Hmm maybe it wasn't a common usage of the name. IIRC some people did call this "currying" in some imperative languages, though never did so in actual FP nor Lisp. You see, clearly this technique here doesn't really have any name. I also think "currying" and "partial application" kinda-sorta make sense. Unbound variables in a "immediate function" are merely hidden parameters of the function if analyzed semantically. The question is whether you pass them by-ref or by-value.
    - jerf 4y ago
      Yes, some people misuse the concept of currying in the wild. It really shouldn't be changed from its FP definition, because it is fundamentally different from partial application. Basically currying is only useful in Haskell. If you don't build it into your compiler as a first-class citizen you're going to be forever fighting a proliferation of closures in your generated code and then trying to optimize them back out. This is not a wise approach to language design. Partial application is the better solution in imperative languages. It is also not relevant to this discussion in the slightest, because the behavior of allocation and how those are passed into closures is independent of the question of how you spell those closures in code. You can easily build currying or some syntactic partial application into Go but leave this problem there, and you can easily fix this problem without building currying or partial application in.
gkfasdfasdf 4y ago
I didn't see it mentioned in the article, but what is the perf cost to this change? Will there be more GC due to new variables every loop iteration?
- jlouis 4y ago
  The reason for this change is to make wrong programs into correct programs. The examples will introduce more GC pressure, because the `append(..)` will append different values as the programmer expected. Thus, correcting the bug adds to GC pressure. In the cases where the new value isn't appended, the compiler can easily optimize it away, and reuse the same memory location in a register or on the stack. My guess is that the optimizer portion of the compiler is at a place nowadays where this will happen, with no further change needed. When the compiler was new, it might have been a regression in efficiency.
deleted 4y ago
[deleted]
lovasoa 4y ago
Ecmascript had the same problem, and fixed it without breaking backwards compatibility. In javascript, for (var i=0; i<3; i++) setTimeout(() => console.log(i)) Would print 3,3,3. So in ecmascript 6, blocked-scope variables were introduced, but the semantics of old-style "var" declarations was not changed. You can now write for (let i=0; i<3; i++) setTimeout(() => console.log(i)) Which prints 0,1,2 as expected. And you don't have to go open a `go.mod` file to know what the code you are reading does.
- dilap 4y ago
  Right call for js, but I think not for C# or Go, since as time goes on, the old behavior will be more and more of a relic, but you'll be paying the cost for having two syntaxes forever. (In js tho you have no alternative, because you don't have anything like go.mod (I think?) and you really can't break backcompat.)
tuukkah 4y ago
I find this same semantic in JavaScript weird albeit practical. Many descriptions of either for-loops or let-variables don't mention that their combination is a special case: each iteration gets its own instance of the variable, but somehow the result of i++ gets copied to the following iterations. Here you can see that the i from the previous iteration gets copied first and i++ applies to the next iteration: for (let i = 0; i < 10; i++) { setTimeout(() => { console.log(i); }, 1000); } // prints 0...9 (as expected?) I always thought of the i++ happening at the end of the previous iteration, but that's wrong as it produces a different result if written explicitly: for (let i = 0; i < 10; ) { setTimeout(() => { console.log(i); }, 1000); i++; } // prints 1...10 (as expected?) (Also, these don't change if I remove the curly braces, so the let is not scoped within curly braces as I thought...)
- roywiggins 4y ago
  My knowledge of javascript semantics is pretty limited, but you can see what's happening more clearly: for (let i = 0; i < 10; ) { setTimeout(() => { console.log(i, foo); }, 1000); let foo = "baz-"+i; i++; } // "1 baz-0", "2 baz-1, etc It's behaving as each loop iteration is creating its own closure, and the inner function is referring to those variables by reference. So any changes you make to them inside the body of the loop will end up visible to the inner function.
  - tuukkah 4y ago
    My point is that for the i++ to make the loop advance (whether it is written inside the block or outside it in the header), behind the scenes, the value of the old i is copied to the new i at the beginning of each iteration. I think to then understand why my two examples produce different results, you have to know that the i++ in the header happens to be executed after that copying has been made (an arbitrary choice?), while the i++ within the body will be (naturally) executed before the copying. I suppose the way it works can be intuitive, but it can also be confusing if you think of the i++ as the last action of each iteration in both of my example cases. Actually, there's a third way to write the example loop, but who can guess which result it gives? for (let i = 0; i++ < 10; ) { setTimeout(() => { console.log(i); }, 1000); } ... it will print 1...10! So even within the loop header, one part is run before the copying and another part after the copying. How is this intuitive and where is this documented apart from the language spec? EDIT: My bad, of course the iteration condition has to be checked at the beginning of the iteration, so in this third example i == 1 during all of the first iteration). Here's the relevant spec section 14.7.4.3 ForBodyEvaluation: https://tc39.es/ecma262/multipage/ecmascript-language-statements-and-declarations.html#sec-forbodyevaluation https://tc39.es/ecma262/multipage/ecmascript-language-statem... There we can finally see that on the first iteration, an environment is created (and iteration variables copied) before the test ("step 2"). After that, a new environment is created (and iteration variables copied) towards the end of each iteration, before the increment step ("step 3.e" and "step 3.f").
ok_dad 4y ago
I'm glad I read this today because I'm starting to do some work in Go. Anyone have any other good sources for "gotchas" in Go (Go-tchas?) that I can read to familiarize myself with how to think better in Go?
- bts 4y ago
  Data Race Patterns in Go: https://eng.uber.com/data-race-patterns-in-go/ https://eng.uber.com/data-race-patterns-in-go/
  - deleted 4y ago
    [deleted]
AtNightWeCode 4y ago
Looks like a bug you would expect to find in JS. ;) Seriously, I think the bug is apparent. But I have done time in C. Closures btw is a sucha horrible pattern. Added in many langs. Always banned by corporate guidelines. The fib example on the GO site is a good example on how confusing it can be. It is right up there with promises and other trash.
arh68 4y ago
This seems reasonable. As others state, this is hardly an issue specific to Go. In CoffeeScript (which I think solves this nicely), it looks something like for i in [0...3] # 3 3 3 setTimeout (() -> console.log "#{i}"), 100 for i in [0...3] # 0 1 2 setTimeout (do (i) -> () -> console.log "#{i}"), 100 for i in [0...3] # 0 1 2 do (i) -> setTimeout (() -> console.log "#{i}"), 100
vagab0nd 4y ago
I used to get compiler warnings for this particular problem (getting a warning tag in Vim, not when compiling). Sadly the warning stopped working after some update and I have been bitten by this several times since.

464 points by denysvitali 4y ago

Yeah… I seem to remember scratching my head because of a variation of this problem. It was kind of driving me nuts. Yet the number of moments like this has been low for me in Go compared to C++.

jupp0r 4y ago

How does this work, in this code: var all []\*Item for _, item := range items { all = append(all, &item) } When &item is the same for all iterations, that means that it's pointing to the same memory address. Is each item in items copied to this address prior to each iteration body invocation? This seems strange as this copy could potentially be very expensive. What am I missing?

silisili 4y ago
Nothing, you got it. Most of the time you're iterating a slice of pointers, though, so only the address gets copied. And in those cases, this bug doesn't exist(unless of course you're going from * to ** for some reason).
foldr 4y ago
To add to the sibling, you could iterate by index to avoid the copy: for i := range items { ptr := &items[i] ... } In this case of course you also get different semantics. The ptr variable is bound to the address of each of the original Item values in the slice, whereas in the code in your comment, &item is the address of a single heap-allocated Item variable.
ytch 4y ago
Could we assume it is like executing: var item Item item = items[0] all = append(all, &item) item = items[1] all = append(all, &item) ...
skywhopper 4y ago
You aren’t missing anything. How expensive the copy is depends on the type of item. This is why you should probably use pointers in slices you’re going to loop over instead of the object itself.

zebracanevra 4y ago

If you're at all interested in how for loops and scope work in Javascript, Jake Archibald and Surma have a great video on the topic: https://www.youtube.com/watch?v=Nzokr6Boeaw https://www.youtube.com/watch?v=Nzokr6Boeaw Similar changes were made in newer versions of ES so that the for loop in this article works out of the box, like C#. Slightly off-topic to this article: I wish "do while" loops had the "while" condition in the inner scope, not the outer scope. So many times I have wished that I could access the inner scope... I end up using a while(true) with an if { break; } at the end instead in 99% of cases where a do while could've been the perfect thing...

chrismorgan 4y ago
Except the ECMAScript changes weren’t changes but additions. Anything using the var keyword gets the traditional semantics, anything using let and const gets lexical scoping. As an example: x = []; for (var i of [1, 2, 3]) { x.push(() => i); } x.map(f => f()) == [3, 3, 3] x = []; for (let i of [1, 2, 3]) { x.push(() => i); } x.map(f => f()) == [1, 2, 3] What’s being proposed for Go is instead a breaking change.
- baq 4y ago
  > What’s being proposed for Go is instead a breaking change. given the context, it might actually be a fixing change...
- foldr 4y ago
  >What’s being proposed for Go is instead a breaking change. It sounds like you only get the breaking change in modules that require above a certain version of Go. So this should not break old code. It's perhaps more analogous to the way that "use strict" in JavaScript 'breaks' parsing of octal constants.
  - chrismorgan 4y ago
    Ah, my apologies; I missed that part. (I last used Go a decade ago, long before go.mod.) Then this seems fairly reasonable, given also how rarely things will depend on one way or the other.

hardware2win 4y ago

How did they even end up with such a crazy behaviour?

mpartel 4y ago
Several languages have or have had this behaviour. C# made a similar change to its 'foreach' loops a long time ago (but not 'for' loops for some reason). I suppose it's the natural way to do it when implementing a language and not thinking about it too much. It makes a for-loop equivalent to a simple while loop with the loop vvariable initialized outside of the loop.
- hardware2win 4y ago
  In c# you mean closure capture?
  - masklinn 4y ago
    Almost certainly, that is the vehicle for the issue in most languages.
- Blikkentrekker 4y ago
  To be fair it is no more difficult to implement it with the variable initialized inside of it. In fact, it's easier to.
- int_19h 4y ago
  The C# change is described in detail in the linked GitHub issue - one of C# devs left a comment explaining it. TL;DR is that both "for" and "foreach" scoping fixes would be breaking changes, but "foreach" was easier to justify because it was already a C#-specific construct syntactically, unlike "for" which uses the same exact syntax as C, Java etc, and they were very sensitive to backwards compatibility at the time (esp. since the tooling didn't have the ability to target various language versions within the same project easily). At the same time, "foreach" represented the vast majority of breakage when they looked at existing code, perhaps because the scoping in classic "for" is more obvious due to the fact that variable mutation is explicit there.

jstimpfle 4y ago

WTF. I haven't been interested in Go after looking at it briefly a decade ago. This post brings back the weird feeling about it that I had then. It looks like taking the address of a local variable (item := item; append(..., &item);) and using that outside the scope of the local variable. But apparently that is ok. Variations of such perceived weirdness exist in many other languages with complicated "object models" as well to be fair. Delphi has some strange adressing stuff going on as well. Python has this weird "default list" thing. Most object languages don't let you take the address of something (like the Go example shows) at all, but have only object references which I find unergonomic.

silon42 4y ago
I also feel there is a problem in that line, not necessarily just a for loop. Maybe that's my C/C++/Rust experience.
- tialaramex 4y ago
  In Rust we know we only gave the append function an immutable reference which lives until the next iteration. If append is OK with that, we're golden. Maybe internally it clones these references -shrug. If append() needs the thing, not just an immutable reference which expires soon, its signature would demand we move one into it, and we don't have one so we'd need to e.g. make one with Clone.
  - maccard 4y ago
    > Maybe internally it clones these references -shrug. That's what go does under the hood in 95% of cases. If I do: func foo(a int) *int { return &a } func main() { fmt.Println(*foo(3)) } It does the right thing. It's just that in this case (and in lambdas) they got it wrong.
  - masklinn 4y ago
    TBF Rust also intentionally used the semantics Go is migrating to from the start, because: - experience with the issue in languages with wider scoping e.g. it’s a common issue in JS, as well as Python (though slightly less so) - Rust’s iterators were originally internal so that was pretty natural (1) is also why for(let and for(const have different scoping than for(var in JS: `var` has function scoping, `let` and `const` were introduced with block scoping, and for loops they were specifically specced with “inner” (per-iteration) scope.

fomine3 4y ago

It's great that who worked for similar changes on C#, responses immediately.

corsix 4y ago

Lua gets this right - the lowering of loops (e.g. https://www.lua.org/manual/5.1/manual.html#2.4.5 https://www.lua.org/manual/5.1/manual.html#2.4.5) says “var is invisible” and has “local v = var”, the latter akin to Go’s “item := item”

benhoyt 4y ago

I'm an experienced Go programmer and still accidentally do this every so often. It certainly bit me and my team several times when we were newish to Go. And particularly for newbies, it's hard to debug and understand when you do run into it. All of which to say I'm really glad they're trying to fix this.

tezza 4y ago
Yep, bitten by this one on repeat myself. definitely a WTF moment each time.
- kubb 4y ago
  same here, happened several times when launching a goroutine inside the loop the fix was usually adding x := x before go func() { do something with x }
  - masklinn 4y ago
    A common alternative fix is to use IIFE, since the expression must be a call. `go` evaluates everything but the final function call in the context of the caller. So go func (x int) { … } (x) Will do the same, and is easier to extract to a named function.
    - Joker_vD 4y ago
      The latter is also easier to re-inline accidentally because go func (x int) { do_work(x) } (x) seems like a very roundabout way to say go do_work(x) The "x := x" solution too suffers from this problem but slightly less: both idioms look like they are no-ops (while they are actually not) but at least "x := x" is weird enough to look like it was a deliberate choice, not some vestige from refactoring.
      - masklinn 4y ago
        Er… it makes no difference? Your first version is in fact a worse way to write the second one.

zendist 4y ago

Thrilled to see Jared Parsons of the C# team pitch in and provide some perspective on how things were done for C#5 when a similar change was made. Kudos Jared!

pjmlp 4y ago
Same here, DevDiv is now polyglot focused, so you will see regular comments from .NET folks on other languages as well (mainly Java and Go). David Fowler tends to tweet every now and then about them as well.
Cthulhu_ 4y ago
Yeah, it's often a joy to read these posts and the perspectives on it; this is how programming language development should be done.
int_19h 4y ago
What's interesting is that C# 5 release (which made the breaking change) was back in 2012, and both the change and the reasons for it were very widely discussed at the time. This is right around the time when Go shipped its 1.0, and it's kinda surprising that they either didn't look closely at "near-peer" languages, or if they did, couldn't see how this problem was fully applicable to their PL design, as well. (Note that C# at least had the excuse of not having closures in the first version, which makes scoping of "foreach" moot - the problem only showed up in C# 2.0. But Go had lambdas from the get-go, so this interaction between loops and closures was always there.)
- pjmlp 4y ago
  Well, this isn't the only language feature where Go designers ignored the path trailed by other languages. Still looking forward to the day they will discover Pascal enumerations.

colonwqbang 4y ago

item := item What does this do? That should be a no-op by default in any sane language.

_ikke_ 4y ago
The := both declares a new variable by an inferred type and assigns a value to it. Explicitly, this would be the same as var item *Item = item It creates a locally scoped variable that shadows the existing one.
- colonwqbang 4y ago
  Ah ok, so different name resolution rules apply on the left and right sides of the operator? On the left side, "item" resolves to the "item" in inner scope (that is declared on the same line). On the right side it resolves to the "item" in outer scope. Then I understand what's happening though I'm still not convinced it's sane behaviour. I would expect all identical names in a single scope to refer to the same variable.
  - deleted 4y ago
    [deleted]
ithkuil 4y ago
item = item That is a no-op a := b Declares a new variable a and assigns it. It's sugar for: var a = b So read it as: var item = item Many languages have scopes and variable shadowing. You can argue this is bad practice or whatever, but it's only confusing if you think := is assignment which is covered in the first page of the Go language tutorial.
- valenterry 4y ago
  Not a go dev, but var item = item should still be a noop imo. If it does some magic like dereferencing, make it explicit. And no, other languages making the same mistake is not a good excuse. Especially not for a language focused on inexperienced newgrads.

the_gipsy 4y ago

> Loop variables being per-loop instead of per-iteration is the only design decision I know of in Go that makes programs incorrect more often than it makes them correct. Since it is the only such design decision, I do not see any plausible candidates for additional exceptions. The bar is quite low.

dontlaugh 4y ago
It's also very subjective. Just off the top of my head I can think of: - zero values instead of sum types - nil, especially on interfaces - goroutines sharing memory - goroutine panics not being raised in the spawner by default
seer-zig 4y ago
golang has quite the suspect decisions that show up when writing large programs https://www.uber.com/blog/data-race-patterns-in-go/ https://www.uber.com/blog/data-race-patterns-in-go/
- cesarb 4y ago
  Working link: https://www.uber.com/en-US/blog/data-race-patterns-in-go/ https://www.uber.com/en-US/blog/data-race-patterns-in-go/ (For me, the link you posted does a 302 redirect to https://www.uber.com/pt-BR/blog/data-race-patterns-in-go/ https://www.uber.com/pt-BR/blog/data-race-patterns-in-go/ which gives me a 404 error page. It's a bit insane that whether the link you posted works or not depends on your locale, and unfortunately this is not the first time I've seen this kind of baffling redirect misbehavior.)

hknmtt 4y ago

Strange, never had a problem with this. If I need to reference non-pointer I just use key(append(foo, items[k])) but i find it weird that the print example is not working since in loop the value is passed by copy, just like everywhere else in go, so it should work correctly. as for pointer, again, pass by copy. I see no problem here, it either never bit me in my ass or i figured it out soon in my Go journey.

bjoli 4y ago

I have said this before, but a source->source compiler is a very handy thing to have. A loop construct in chez or guile scheme would probably translate into something tail recursive. You could then macro-expand and optimize the loop to the code that is then compiled into lower level. I have been in situations where I have had to expand a macro to figure out what is going on. Not having it in a situation like this (where a for loop is obviously just a goto or a tail call) is usuallly a pain in the ass. If it was translated into the same language it would also be easier to define what it should translate to.

valenterry 4y ago
I'd rather use a language from the beginning that isn't made up accommodate tons of newgrads as fast as possible on a big codebase. That is what Google needs, but most people are better of with more advanced languages. But people follow Google just like they did with Angular. I predict Go will follow the same destiny, will just take longer. The tech debt is still piling up.

shp0ngle 4y ago

If you never did a mistake connected to this and caused a bug in real prod code, you haven’t been writing go for long enough.

barsonme 4y ago
Ehhhh, I’ve been writing Go for ~10 years and I’ve never had this make it into prod. Many times in local development and testing, though.
Traubenfuchs 4y ago
Ideally at least one of the tests out of your testing pyramid would catch something like that though.
- Cthulhu_ 4y ago
  Ideally yeah, but unfortunately very few people work in ideal circumstances.
bandrami 4y ago
I internalized early on that the likelihood of this mistake means Go wants me to refactor when I find myself liable to making it.
ainar-g 4y ago
Or you've been using the looppointer analyser[1] from the start, heh. [1]: https://github.com/kyoh86/looppointer https://github.com/kyoh86/looppointer
viiralvx 4y ago
Yeahhhhh, I worked at a big company on their CI product and I had this bug bite me and break builds for a little bit. Luckily caught my mistake and fixed it within 15 minutes, but it's always easy to overlook.

bborud 4y ago

That was a very well written issue.

mikepurvis 4y ago
Shockingly high quality discussion too— have the comment moderation tools on Github gotten better in recent years, or is this just the golang community being awesome?

xmprt 4y ago

I'm all for this change. I've had to explain this weird programming quirk of Go to coworkers on multiple occasions and can't wait for it to be solved.

Cthulhu_ 4y ago
I'm just happy that it's one of the few weird quirks of Go. In my spheres, Javascript is the most commonly used language, and the 'wat' presentation is frequently referenced to.

mftb 4y ago

I'm happy the Go team takes breaking changes seriously. They should go ahead and fix this. The current semantics are lame.

anotherhacker0 4y ago

Anyone know the status of overall golang 2? also anyone know of any proposals around generics or pattern matching?

mseepgood 4y ago
There are no plans for a literal version 2 anymore. Generics were released in March. There are no active proposals for pattern matching.
- gus_leonel 4y ago
  A quick guide to the generics support that's been added to Go: https://bitfieldconsulting.com/golang/generics https://bitfieldconsulting.com/golang/generics
- jerf 4y ago
  Specifically, Golang 2.0 was reserved for backwards incompatible changes, generally understood to be of some significant size, not twiddly bits like this that are adequately covered by a go.mod flag. Generics was thought to possibly involve backwards incompatible changes, but it was done without backwards incompatibility. There aren't any similarly-sized issues on the horizon right now that would result in a 2.0. If you track the issues closely on GitHub, there's a constant low-level discussion going on about it, but so far nothing seems to be both important enough and large enough to justify the version change, like, not even close.

tgv 4y ago

I'm absolutely not looking forward to potentially breaking changes. Stability is such an important selling point (*). This is not something that can be fixed by a search-replace or a refactoring operation. And it's just begging for problems in the community. (*) please don't reply with "then you should keep the version below 1.20"

mananaysiempre 4y ago
Given a tool sophisticated enough to track variable scope (not a tall order), this absolutely can be fixed mechanically.
masklinn 4y ago
> please don't reply with "then you should keep the version below 1.20" That is literally the solution. Just remove your go.mod or keep it under 1.20, and you’ll keep the old behaviour forever. It’s the same as e.g. Rust’s edition, if you hate your life you can keep using Rust 2015 semantics to this day, and forever.
- tgv 4y ago
  I knew there would be someone. So if you want a feature in a higher version, update your code base. Yeah, that's what stability doesn't mean. And "it's your problem now" isn't great Go advocacy either.
  - masklinn 4y ago
    There is no feature, it only sets the static evaluation of the language itself. Think for a second, if the go.mod stricture locked in the entire thing, you’d just be told to not upgrade. The entire point of the system is that it’s possible to update language-level semantics while allowing for the rest to progress for everybody. Setting go.mod means you get stdlib updates, but the semantics of the for loop does not change. And if the designers decide to add similar breaking changes in the future (change string literals or whatever), you’ll also be ignoring those.

crnkofe 4y ago

I gave this post all the upvotes and hearts I got. It's one of those weird Go language quirks where developer expectations just never match what's going on behind the scene. I still often find myself always looking at the loops if they actually modify/use correct variables.

zadwang 4y ago

If I understand correctly the fix requires the new code to add a line in go.mod to use the new behavior. This is about the same as adding x:=x in the loop, and more hidden. Not good. The alternative can be that whenever address of iteration variable is used inside the loop the variable is per iteration and otherwise it is per loop. This way it is not breaking the old code and have new semantics.

zimpenfish 4y ago
> whenever address of iteration variable is used That wouldn't fix the closure problem where the address isn't taken.
- zadwang 4y ago
  You are right. Would it be possible to add closure use as another case?
  - zimpenfish 4y ago
    I guess you could but now you have to special-cases that need static analysis and all that jazz. Probably easier to just put it behind a version flag.
viraptor 4y ago
> This is about the same as adding x:=x in the loop It's one per module though. In large enough modules you'll have tens of x:=x though. I assume this also opens the option of doing more such changes through the same system in the future.
- masklinn 4y ago
  And assuming new projects get the new mod by default, it also makes the langage default-safe, as opposed to requiring the cognitive overhead of evaluating individual loops, or requiring this nonsense as explicit prolog to every loop.
- 4y ago

tialaramex 4y ago

> we gave the general rule that language redefinitions like what I just described are not permitted Not just a "general rule" that document also specifically talks about precisely this issue (for loops) and resolves that Go will not fix this. Change is the only constant, we should design systems with the expectation that they'll need to adapt over time or they will be replaced by something which can. With this mindset, Go should have solved the for loop problem years ago, just as C# did. This could have been a story about how once upon a time Go had these very silly for loop semantics, but that hasn't been true for many years. This necessity of change is why I think the decision not to take Epochs for C++ 20 was much more consequential than things like rejecting the "Goals and priorities" paper which had immediate effects (in that particular case spurring the Carbon experiment).

morelisp 4y ago
Arguably if Go had done this years ago, go.mod would not have existed and this would have an even bigger impact. Just like we'll rarely pick exactly the right behavior, we'll rarely pick exactly the right time to fix the broken behavior.
- varispeed 4y ago
  How one depends on another? To me it sounds these things are orthogonal.
  - masklinn 4y ago
    Because of time? How would you use go.mod when go mod does not exist?
  - room271 4y ago
    go.mod provides a way to introduce the new behaviour incrementally. The key point is the dependencies can (and already do) declare different go versions in their go.mod file. When everything is compiled, each module is compiled with the behaviour that applies to its own declared go version. So even if you update your go.mod go version to take advantage of this (soon to be) new behaviour, you can continue to use existing deps happily without worrying they will break.

BoppreH 4y ago

The same problem frequently bites me in Python too: functions = [] for i in range(10): functions.append(lambda: print(f'Hello {i}')) for fn in functions: fn() # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 I find it one of Python's biggest warts because it's silent, hard to troubleshoot (especially the first time!), and like in Go the most straightforward fix looks like a mistake (i=i): for i in range(10): functions.append(lambda i=i: print(f'Hello {i}')) I guess this is a lesson in designing language semantics that match people's intuitions, and learning from previous languages' mistakes.

Simplicitas 4y ago
Yup. "Match [human] intuition" is the operative phrase in all of this. Thanks. Hope Go addresses this soon.
Semaphor 4y ago
If I do the same with for instead of foreach in C#, my IDE gives me a warning: "Captured variable is modified in the outer scope". Isn’t that the case for Python?
- ketzu 4y ago
  Copy pasting OPs code to pycharm: No warning.
  - Semaphor 4y ago
    Probably worthy of a YouTrack report ;)
  - jstx1 4y ago
    Pylint catches it - https://pylint.pycqa.org/en/latest/user_guide/messages/warning/cell-var-from-loop.html https://pylint.pycqa.org/en/latest/user_guide/messages/warni...
- maxloh 4y ago

foldl2022 4y ago

Glad that Nim had solved this problem with a macro `capture`. It's up to the developers to capture loop variables or not.

brudgers 4y ago

Oh, schadenfreude of Turing's Tarpit I giggle a bit http://www.lispworks.com/documentation/lw51/CLHS/Body/m_loop.htm http://www.lispworks.com/documentation/lw51/CLHS/Body/m_loop...

assbuttbuttass 4y ago
Lisp suffers from this same defect, maybe that was your point? What does the following code print out? (defvar refs (loop for i from 1 to 10 collect (lambda () (format t "~d~%" i)))) (loop for f in refs do (funcall f))

thiht 4y ago

I fail to see any downside to changing this semantics. This has honestly always felt like a language design bug more than anything. Having to write foo := foo at the beginning of a loop for it to behave as expected is a strong design smell. The gradual breaking (of fix depending on your point of view) with explicit opt-in looks great to me.

esjeon 4y ago

Am I the only one who thought about currying here? I mean, neither the ticket nor the thread mentions it. I don't think it's such an alien concept to people. TBH, with my full sincerity, this isn't really a way to solve the problem. Currying has been used for decades for reasons. For-loop isn't the only place where you introduce outer-scope variables into an immediate function. You can always have other variables involved, and you'll still have to be careful. Even though this change will reduce the absolute number of bugs, people will still have the same issue.

justinpombrio 4y ago
Currying is the observation that a function that takes a pair as an argument, `(a, b) -> c`, is equivalent to a function that returns a function: `a -> (b -> c)`. What's that have to do with for loops?
quietbritishjim 4y ago
Are you sure you mean currying? Currying is where you transform a function taking multiple arguments into a function that only takes one of the arguments and returns another function which takes one of the remaining arguments which (...recursively). Maybe you meant partial function application? Even that doesn't seem relevant, given that the function is actually being called immediately in the first example.
- esjeon 4y ago
  Hmm maybe it wasn't a common usage of the name. IIRC some people did call this "currying" in some imperative languages, though never did so in actual FP nor Lisp. You see, clearly this technique here doesn't really have any name. I also think "currying" and "partial application" kinda-sorta make sense. Unbound variables in a "immediate function" are merely hidden parameters of the function if analyzed semantically. The question is whether you pass them by-ref or by-value.
  - jerf 4y ago
    Yes, some people misuse the concept of currying in the wild. It really shouldn't be changed from its FP definition, because it is fundamentally different from partial application. Basically currying is only useful in Haskell. If you don't build it into your compiler as a first-class citizen you're going to be forever fighting a proliferation of closures in your generated code and then trying to optimize them back out. This is not a wise approach to language design. Partial application is the better solution in imperative languages. It is also not relevant to this discussion in the slightest, because the behavior of allocation and how those are passed into closures is independent of the question of how you spell those closures in code. You can easily build currying or some syntactic partial application into Go but leave this problem there, and you can easily fix this problem without building currying or partial application in.

gkfasdfasdf 4y ago

I didn't see it mentioned in the article, but what is the perf cost to this change? Will there be more GC due to new variables every loop iteration?

jlouis 4y ago
The reason for this change is to make wrong programs into correct programs. The examples will introduce more GC pressure, because the `append(..)` will append different values as the programmer expected. Thus, correcting the bug adds to GC pressure. In the cases where the new value isn't appended, the compiler can easily optimize it away, and reuse the same memory location in a register or on the stack. My guess is that the optimizer portion of the compiler is at a place nowadays where this will happen, with no further change needed. When the compiler was new, it might have been a regression in efficiency.

deleted 4y ago

[deleted]

lovasoa 4y ago

Ecmascript had the same problem, and fixed it without breaking backwards compatibility. In javascript, for (var i=0; i<3; i++) setTimeout(() => console.log(i)) Would print 3,3,3. So in ecmascript 6, blocked-scope variables were introduced, but the semantics of old-style "var" declarations was not changed. You can now write for (let i=0; i<3; i++) setTimeout(() => console.log(i)) Which prints 0,1,2 as expected. And you don't have to go open a `go.mod` file to know what the code you are reading does.

dilap 4y ago
Right call for js, but I think not for C# or Go, since as time goes on, the old behavior will be more and more of a relic, but you'll be paying the cost for having two syntaxes forever. (In js tho you have no alternative, because you don't have anything like go.mod (I think?) and you really can't break backcompat.)

tuukkah 4y ago

I find this same semantic in JavaScript weird albeit practical. Many descriptions of either for-loops or let-variables don't mention that their combination is a special case: each iteration gets its own instance of the variable, but somehow the result of i++ gets copied to the following iterations. Here you can see that the i from the previous iteration gets copied first and i++ applies to the next iteration: for (let i = 0; i < 10; i++) { setTimeout(() => { console.log(i); }, 1000); } // prints 0...9 (as expected?) I always thought of the i++ happening at the end of the previous iteration, but that's wrong as it produces a different result if written explicitly: for (let i = 0; i < 10; ) { setTimeout(() => { console.log(i); }, 1000); i++; } // prints 1...10 (as expected?) (Also, these don't change if I remove the curly braces, so the let is not scoped within curly braces as I thought...)

roywiggins 4y ago
My knowledge of javascript semantics is pretty limited, but you can see what's happening more clearly: for (let i = 0; i < 10; ) { setTimeout(() => { console.log(i, foo); }, 1000); let foo = "baz-"+i; i++; } // "1 baz-0", "2 baz-1, etc It's behaving as each loop iteration is creating its own closure, and the inner function is referring to those variables by reference. So any changes you make to them inside the body of the loop will end up visible to the inner function.
- tuukkah 4y ago
  My point is that for the i++ to make the loop advance (whether it is written inside the block or outside it in the header), behind the scenes, the value of the old i is copied to the new i at the beginning of each iteration. I think to then understand why my two examples produce different results, you have to know that the i++ in the header happens to be executed after that copying has been made (an arbitrary choice?), while the i++ within the body will be (naturally) executed before the copying. I suppose the way it works can be intuitive, but it can also be confusing if you think of the i++ as the last action of each iteration in both of my example cases. Actually, there's a third way to write the example loop, but who can guess which result it gives? for (let i = 0; i++ < 10; ) { setTimeout(() => { console.log(i); }, 1000); } ... it will print 1...10! So even within the loop header, one part is run before the copying and another part after the copying. How is this intuitive and where is this documented apart from the language spec? EDIT: My bad, of course the iteration condition has to be checked at the beginning of the iteration, so in this third example i == 1 during all of the first iteration). Here's the relevant spec section 14.7.4.3 ForBodyEvaluation: https://tc39.es/ecma262/multipage/ecmascript-language-statements-and-declarations.html#sec-forbodyevaluation https://tc39.es/ecma262/multipage/ecmascript-language-statem... There we can finally see that on the first iteration, an environment is created (and iteration variables copied) before the test ("step 2"). After that, a new environment is created (and iteration variables copied) towards the end of each iteration, before the increment step ("step 3.e" and "step 3.f").

ok_dad 4y ago

I'm glad I read this today because I'm starting to do some work in Go. Anyone have any other good sources for "gotchas" in Go (Go-tchas?) that I can read to familiarize myself with how to think better in Go?

bts 4y ago
Data Race Patterns in Go: https://eng.uber.com/data-race-patterns-in-go/ https://eng.uber.com/data-race-patterns-in-go/
- deleted 4y ago
  [deleted]

AtNightWeCode 4y ago

Looks like a bug you would expect to find in JS. ;) Seriously, I think the bug is apparent. But I have done time in C. Closures btw is a sucha horrible pattern. Added in many langs. Always banned by corporate guidelines. The fib example on the GO site is a good example on how confusing it can be. It is right up there with promises and other trash.

arh68 4y ago

This seems reasonable. As others state, this is hardly an issue specific to Go. In CoffeeScript (which I think solves this nicely), it looks something like for i in [0...3] # 3 3 3 setTimeout (() -> console.log "#{i}"), 100 for i in [0...3] # 0 1 2 setTimeout (do (i) -> () -> console.log "#{i}"), 100 for i in [0...3] # 0 1 2 do (i) -> setTimeout (() -> console.log "#{i}"), 100

vagab0nd 4y ago

I used to get compiler warnings for this particular problem (getting a warning tag in Vim, not when compiling). Sadly the warning stopped working after some update and I have been bitten by this several times since.