4 ms·

The weird of function-local types in Rust

116 points by lukastyrychtr 2y ago

jerf 2y ago
I have found, across several languages I've used, that types embedded into functions are generally a bad idea, and I think the general principle is that types generally end up needing to be exposed to any code that will also test that code. So, for instance, it's fine to confine types to some particular module, as long as those types are internal-only, but confining them within functions generally becomes a bad idea. I know the complaints many of you are gearing up to type, but my statement is a bit more complicated than you may have realized on first read; the key is the word "becoming", that I'm looking at the lifetime of the code and not a snapshot. The problem with embedding types into those smaller scopes is that while it may work at first... of course it does, it compiles, right?... they become an impediment to a number of operations over time. First, as I mentioned, testing is very likely at some point over the evolution of the module to want to either provide input or examine output, intermediate or otherwise, that exists in those types. Second, as the code grows, you want to be able to refactor things freely, and types embedded in functions form a barrier to refactoring because to refactor you'll have to do something to expose that type now to multiple functions. You do not want barriers to refactoring. Barriers to refactoring are a bigger expense over the long term than any small local gain from putting a type here instead of there, especially when anyone should have "Jump to Definition" readily available in this post-LSP era. Considered over time, over the evolution of the code base, I've just never had any super-local types like this "survive". Every time I think I've found an exception, I've either had my test code or the desire to refactor force me to lift it to the module level. So I just start there now. To the extent there is an exception, testing-only code may be. Testing-only code has very different constraints than production code anyhow. Even then, though, I still find that refactoring problem arises, and test code needs to be refactorable too. On the plus side, while I label them "a bad idea", they are not a "bad idea" that destroys your code base or anything. On the grand scale of "bad ideas" in code, this is down in the "inconvenience" part of the scale. It is almost self-evidently not some sort of disaster and I am not claiming it is. You can always lift it out and move on. But it is one of the many little hygiene habits that add up that helps keep code fluid and refactoring always available to me at a minimum activation-energy cost, because that is really important. (This applies specifically to types that you explicitly define. You can in Haskell, for instance, bash a new type together anywhere simply by creating a tuple (x, y). But this doesn't trigger what I'm talking about because any other bit of the code can bash the exact same type together simply by creating another tuple of the same type, and they'll unify just fine without having to share a type definition in common. No impediment of any kind is created by a new tuple type in that language.)
- chowells 2y ago
  Sometimes I think I want local types in Haskell. I'm creating some new type and some instances for it solely for this function, why does it need global scope? Then I get around to remembering module scope is the actual important thing in Haskell, make sure that type isn't exported from the primary API modules, and get on with life.
  - saghm 2y ago
    Yeah, I think it's pretty rare to actually need a "local" type as opposed to just making it private to a module in Rust as well. The use case the article gives is one of the few where it I could see it being useful; if you're using a macro in the body of a function and want it to generate a type, there's not anywhere else you can define it (without an additional macro invocation outside the function, which often defeats the purpose of trying to wrap up all the boilerplate into one concise place).
- Joker_vD 2y ago
  One of the somewhat useful (but still mind-boggling) uses of local types I've encountered was in Go, writing a custom unmarshaller: func (s *MyLovelyStruct) UnmarshalJSON(b []byte) error { err := json.Unmarshal(b, s) if err != nil { return err } return validateAndMassageMyLovelyStruct(s) } I want to re-use the default "dump JSON key/values into the struct's fields" logic, then add something on top of it. But as written, this method will blow up with stack overflow because json.Unmarshal(b, s) will call s.UnmarshalJSON(b), if it can. So what you can do is this: func (s *MyLovelyStruct) UnmarshalJSON(b []byte) error { type IncognitoStruct MyLovelyStruct tmp := IncognitoStruct{} err := json.Unmarshal(b, &tmp) if err != nil { return err } *s = MyLovelyStruct(tmp) return validateAndMassageMyLovelyStruct(s) } The IncognitoStruct, even if it has exact same fields as MyLovelyStruct (and is castable to it), does not have any of its methods, so json.Unmarshal(b, &tmp) does not recursively call this UnmarshalJSON() method. But even that uses type aliases, not the proper, data-holding, types themselves. I never found any motivation to use those; the package-local types are quite enough.
dathinab 2y ago
> So there is just no way to refer to the User struct outside of the function scope, right?... no matter what tricks you come up with, treat it as that (in case of it being associated to a type treat it as a anonymous type accidentally expose) also please _never_ place a module in a function, for various subtle reasons it's technically possible but you really really should not do it I mean in general limit what items (types, impl blocks) you place in function to very limited cases. If you have a type complex enough so that you need a builder defined in a function you are definitely doing something wrong I think. > Does this mean generating child modules for privacy in macros is generally a bad idea? It depends... IMHO if we look at derive like macro usage, yes it's always a bad idea. Derive like thinks should mainly generate impl blocks, if it really really is necessary types and only if there really is no other way modules. Furthermore they should if possible not introduce any of this in the scope. E.g. it's a not uncommon pattern to place all generated code in `const _:() = {/here/};` which is basically a trick/hack to create a new scope similar to a function scope into which you can place items (functions, imports, types, impl blocks) without polluting the parent scope (and yes that doesn't work for modules they are always scoped by other modules). So does that mean the builder derive does it all wrong? I don't think so sometimes you need to do bad decisions because there are no good solutions.
noelwelsh 2y ago
This seems like an oversight in the design of Rust. I would think that each function call should create a distinct function-local type, so the trick they use to extract the type from the function shouldn't work. I think what's needed is path-dependent types [1] as found in Scala. [1]: http://lampwww.epfl.ch/~amin/dot/fpdt.pdf http://lampwww.epfl.ch/~amin/dot/fpdt.pdf
- GrantMoyer 2y ago
  Rust generally uses lexical scoping, and each function/closure has a unique (possibly anonymous) type per definition, not a type per call. I would therefore expect local types to be per definition too, so the behavior seems fine to me.
  - noelwelsh 2y ago
    I disagree. If I define two modules mod one { struct Cat { name: String } } and mod two { struct Cat { name: String } } I have two distinct types called Cat which are not equivalent. one::Cat != two::Cat // doesn't compile but illustrates the point, I hope Similarly, when I call a function I create a new environment (think, stack frame) for each call which contains values that are distinct from all other calls. I would expect the same to hold for types defined within a function.
    - masklinn 2y ago
      > Similarly, when I call a function I create a new environment (think, stack frame) for each call which contains values that are distinct from all other calls. I would expect the same to hold for types defined within a function. Rust types are not runtime objects. Also just because a function call creates a new environment doesn't mean everything is part of that environment. `static` items are singletons, even if defined within a function (which is a common case when the function should be the only thing directly interacting with the static).
    - GrantMoyer 2y ago
      I don't think the analogy to modules is quite right. I think that maps better to: fn foo() {struct Cat;} fn bar() {struct Cat;} and foo::Cat != bar::Cat. Whereas the a single function with a local type maps better to: mod foo {struct Cat;} mod bar {pub use ::foo::Cat;} mod baz {pub use ::foo::Cat;} and bar::Cat does equal baz::Cat. But maybe I only think that construct maps better because I'm predisposed to the interpetation I described. I do see what your saying, and agree that Rust could work that way; I'm just not convinced it's a bug that it doesn't. The behavior you describe would be more surprising to me than the existing behavior, but clearly that's not a universal sentiment, and I'm not sure which behavior would be less surprising to most people.

116 points by lukastyrychtr 2y ago

jerf 2y ago

I have found, across several languages I've used, that types embedded into functions are generally a bad idea, and I think the general principle is that types generally end up needing to be exposed to any code that will also test that code. So, for instance, it's fine to confine types to some particular module, as long as those types are internal-only, but confining them within functions generally becomes a bad idea. I know the complaints many of you are gearing up to type, but my statement is a bit more complicated than you may have realized on first read; the key is the word "becoming", that I'm looking at the lifetime of the code and not a snapshot. The problem with embedding types into those smaller scopes is that while it may work at first... of course it does, it compiles, right?... they become an impediment to a number of operations over time. First, as I mentioned, testing is very likely at some point over the evolution of the module to want to either provide input or examine output, intermediate or otherwise, that exists in those types. Second, as the code grows, you want to be able to refactor things freely, and types embedded in functions form a barrier to refactoring because to refactor you'll have to do something to expose that type now to multiple functions. You do not want barriers to refactoring. Barriers to refactoring are a bigger expense over the long term than any small local gain from putting a type here instead of there, especially when anyone should have "Jump to Definition" readily available in this post-LSP era. Considered over time, over the evolution of the code base, I've just never had any super-local types like this "survive". Every time I think I've found an exception, I've either had my test code or the desire to refactor force me to lift it to the module level. So I just start there now. To the extent there is an exception, testing-only code may be. Testing-only code has very different constraints than production code anyhow. Even then, though, I still find that refactoring problem arises, and test code needs to be refactorable too. On the plus side, while I label them "a bad idea", they are not a "bad idea" that destroys your code base or anything. On the grand scale of "bad ideas" in code, this is down in the "inconvenience" part of the scale. It is almost self-evidently not some sort of disaster and I am not claiming it is. You can always lift it out and move on. But it is one of the many little hygiene habits that add up that helps keep code fluid and refactoring always available to me at a minimum activation-energy cost, because that is really important. (This applies specifically to types that you explicitly define. You can in Haskell, for instance, bash a new type together anywhere simply by creating a tuple (x, y). But this doesn't trigger what I'm talking about because any other bit of the code can bash the exact same type together simply by creating another tuple of the same type, and they'll unify just fine without having to share a type definition in common. No impediment of any kind is created by a new tuple type in that language.)

chowells 2y ago
Sometimes I think I want local types in Haskell. I'm creating some new type and some instances for it solely for this function, why does it need global scope? Then I get around to remembering module scope is the actual important thing in Haskell, make sure that type isn't exported from the primary API modules, and get on with life.
- saghm 2y ago
  Yeah, I think it's pretty rare to actually need a "local" type as opposed to just making it private to a module in Rust as well. The use case the article gives is one of the few where it I could see it being useful; if you're using a macro in the body of a function and want it to generate a type, there's not anywhere else you can define it (without an additional macro invocation outside the function, which often defeats the purpose of trying to wrap up all the boilerplate into one concise place).
Joker_vD 2y ago
One of the somewhat useful (but still mind-boggling) uses of local types I've encountered was in Go, writing a custom unmarshaller: func (s *MyLovelyStruct) UnmarshalJSON(b []byte) error { err := json.Unmarshal(b, s) if err != nil { return err } return validateAndMassageMyLovelyStruct(s) } I want to re-use the default "dump JSON key/values into the struct's fields" logic, then add something on top of it. But as written, this method will blow up with stack overflow because json.Unmarshal(b, s) will call s.UnmarshalJSON(b), if it can. So what you can do is this: func (s *MyLovelyStruct) UnmarshalJSON(b []byte) error { type IncognitoStruct MyLovelyStruct tmp := IncognitoStruct{} err := json.Unmarshal(b, &tmp) if err != nil { return err } *s = MyLovelyStruct(tmp) return validateAndMassageMyLovelyStruct(s) } The IncognitoStruct, even if it has exact same fields as MyLovelyStruct (and is castable to it), does not have any of its methods, so json.Unmarshal(b, &tmp) does not recursively call this UnmarshalJSON() method. But even that uses type aliases, not the proper, data-holding, types themselves. I never found any motivation to use those; the package-local types are quite enough.

dathinab 2y ago

> So there is just no way to refer to the User struct outside of the function scope, right?... no matter what tricks you come up with, treat it as that (in case of it being associated to a type treat it as a anonymous type accidentally expose) also please _never_ place a module in a function, for various subtle reasons it's technically possible but you really really should not do it I mean in general limit what items (types, impl blocks) you place in function to very limited cases. If you have a type complex enough so that you need a builder defined in a function you are definitely doing something wrong I think. > Does this mean generating child modules for privacy in macros is generally a bad idea? It depends... IMHO if we look at derive like macro usage, yes it's always a bad idea. Derive like thinks should mainly generate impl blocks, if it really really is necessary types and only if there really is no other way modules. Furthermore they should if possible not introduce any of this in the scope. E.g. it's a not uncommon pattern to place all generated code in `const _:() = {/here/};` which is basically a trick/hack to create a new scope similar to a function scope into which you can place items (functions, imports, types, impl blocks) without polluting the parent scope (and yes that doesn't work for modules they are always scoped by other modules). So does that mean the builder derive does it all wrong? I don't think so sometimes you need to do bad decisions because there are no good solutions.

noelwelsh 2y ago

This seems like an oversight in the design of Rust. I would think that each function call should create a distinct function-local type, so the trick they use to extract the type from the function shouldn't work. I think what's needed is path-dependent types [1] as found in Scala. [1]: http://lampwww.epfl.ch/~amin/dot/fpdt.pdf http://lampwww.epfl.ch/~amin/dot/fpdt.pdf

GrantMoyer 2y ago
Rust generally uses lexical scoping, and each function/closure has a unique (possibly anonymous) type per definition, not a type per call. I would therefore expect local types to be per definition too, so the behavior seems fine to me.
- noelwelsh 2y ago
  I disagree. If I define two modules mod one { struct Cat { name: String } } and mod two { struct Cat { name: String } } I have two distinct types called Cat which are not equivalent. one::Cat != two::Cat // doesn't compile but illustrates the point, I hope Similarly, when I call a function I create a new environment (think, stack frame) for each call which contains values that are distinct from all other calls. I would expect the same to hold for types defined within a function.
  - masklinn 2y ago
    > Similarly, when I call a function I create a new environment (think, stack frame) for each call which contains values that are distinct from all other calls. I would expect the same to hold for types defined within a function. Rust types are not runtime objects. Also just because a function call creates a new environment doesn't mean everything is part of that environment. `static` items are singletons, even if defined within a function (which is a common case when the function should be the only thing directly interacting with the static).
  - GrantMoyer 2y ago
    I don't think the analogy to modules is quite right. I think that maps better to: fn foo() {struct Cat;} fn bar() {struct Cat;} and foo::Cat != bar::Cat. Whereas the a single function with a local type maps better to: mod foo {struct Cat;} mod bar {pub use ::foo::Cat;} mod baz {pub use ::foo::Cat;} and bar::Cat does equal baz::Cat. But maybe I only think that construct maps better because I'm predisposed to the interpetation I described. I do see what your saying, and agree that Rust could work that way; I'm just not convinced it's a bug that it doesn't. The behavior you describe would be more surprising to me than the existing behavior, but clearly that's not a universal sentiment, and I'm not sure which behavior would be less surprising to most people.