6 ms·

Interface Upgrades in Go (2014)

88 points by panic 2y ago

McMini 2y ago
While the article highlights interface upgrades, it overlooks potential downsides like increased complexity in debugging. Anyone experienced issues with this?
PUSH_AX 2y ago
I remember working in a team that used interfaces for a ton of things. Every single one of those things only ever had one concrete implementation, still to this day. A lot of those things were also purely for mocking in tests too. Today I don’t use them, unless I need it retrospectively. Which I find is rare. This is not a knock on interfaces as a language feature. Just inserting a random anecdote about pragmatism..
- radicalbyte 2y ago
  So you don't test the parts of your system which need mocks to be effectively tested? How do you avoid the maintenance nightmare that causes? I'm working on a Golang project where we've not yet introduced those kind of tests and, although it can be quicker to code initially, it's slower overall. The project relies heavily on other Open Source systems so the Golang code isn't even that complex. I can't imagine doing something hard and keeping it working over time without being to isolate dependencies to enable testing and keep the structure clear overall.
  - PUSH_AX 2y ago
    I’m saying we did test parts that required mocks, and we used interfaces as a vehicle to streamline that. Rightly or wrongly.
    - PUSH_AX 2y ago
      I’ve just realised you understood and we’re asking present tense I think. The truth is I also do less unit testing, identifying instead where I know confidence is needed and where it’s not.
  - the_gipsy 2y ago
    The question is, are those interfaces used for anything but mocking tests? If not, could there be a better solution?
    - valzam 2y ago
      I keep hearing this but I don't quite understand what the alternative is? I get that maybe introducing interfaces within your own business domain CAN be overused but certainly any external depedency should be fronted by an interface so the rest of the code can be tested without spinning up test http servers, database containers etc. all over the place. It just feels like a really lightweight, simnple thing to have an interface for some API that you are calling. It also means you are less depedent on the concrete implementation of the API and don't leak types and behaviour from systems you don't control into your codebase
      - afavour 2y ago
        This is a situation where I enjoy coding in TypeScript. You don’t need to front with interfaces, you just mock objects that map 1:1 with the expected API. So you write your code targeting the standard Fetch API then just pass in a mock when testing.
      - ashishmax31 2y ago
        +1 to this. Especially with go, most of the (non-integration) tests I end up writing are behaviour driven tests. I'm not really sure how one can write these sort of tests without interfaces and mocks.
        randomdata 2y ago
        One could use build tags to provide alternative implementations over a concrete type, I suppose. But like with everything in life, there is no right solution. Just different tradeoffs. If interfaces best align with the trades you are willing to make, go for it.
      - dgb23 2y ago
        Instead of pushing the external, side-effecting thing down the stack, I put it up the stack and call the logic that generates the data for it from there. Now I don't need to mock an interface, I just need to test data in/out. If you do this consistently, then you have much of the side effecting code residing next to each other per process/input handler etc. It's not always feasible or the right thing. But it's a good default way of structuring code and makes testing much more straight forward. And it makes more clear "at a glance" what happens because you can reason more locally about the side effecting parts. > It also means you are less depedent on the concrete implementation of the API and don't leak types and behaviour from systems you don't control into your codebase You are always dependent on that. Whether you hide it down the stack or lift it up, you still need to put your data into a specific shape, do the same essential checks etc.
- jen20 2y ago
  Something one often sees in Go (which is in my mind an anti-pattern) is the supplier of an object also supplying an equivalent interface, as they would in Java or another nominative adoption system. In Go, the interface belongs to the consumer!
  - slekker 2y ago
    But how do you avoid duplicating the interface in many places? We've tried both approaches, declaring the interface in one package and importing it where used but also declaring it directly colocated with the consumer. I've found the first approach worked better when we had to add or modify the interface methods
    - jen20 2y ago
      You don’t. Each consumer of something declares what it needs via an interface, something concrete may be passed that implements it - including fakes for tests. Of course this doesn’t apply when using interfaces for polymorphic return types in a library, but that is a distinct use case for interfaces to address. Perhaps some of this is just taste, but this is the way I’ve found interfaces in Go to be most useful.
nickcw 2y ago
I used to use interface upgrades in rclone (a program which syncs your data to many different cloud providers) for all the optional features a backend (cloud provider) might have, for example renaming a file. However the proxy problem became unmanageable. That's when I had a backend (eg the crypt backend which encrypts any other backend) which wraps another backend. Can the crypt backend rename things? That depends on what it is wrapping and you'd have to upgrade the interface call it and then get a special error to find out. Eventually I switched to a table of bound method pointers which were easy to test against nil to see whether they were implemented and could be filled with a very small amount of reflection. In my experience interface upgrades are useful but the proxy problem is very real so use sparingly only! I did suggest at one point (to rsc) an addition to Go which would allow interface proxies to take methods away from their method set at run time to fix this problem, so the proxy could only have the methods it could implement. I've no idea how difficult this would be to implement though.
- lazypenguin 2y ago
  Thanks for rclone, great tool!
- memset 2y ago
  Could you share a very small code snippet describing what you mean by bound method pointers? (Huge fan of rclone!)
  - jerf 2y ago
    Something like this: https://go.dev/play/p/AVVz0qs2i1e https://go.dev/play/p/AVVz0qs2i1e This is much less convenient than using interfaces directly when that is possible, but comes with a lot more power. You have more insight into the "interface" value, you can use this to manually implement prototype-like behavior as seen in Javascript, you can deliberately make an object where one "method" is invoked on one object and one "method" is invoked on another, which is hard to read but may be just what you need in some situation, etc. There are a lot of ways to improvise on this tune. Also note most OO or OO-flavored languages can do this, Go has no special claim to it. But the tradeoffs are pretty similar in all those languages... you want to be sure this is what you need before you use it, because the design costs are significant.
rollulus 2y ago
Note that this is not necessarily a great thing to overuse. Also note that the article is 10 years old. Relying on such upgrades sort of introduces a dark and fuzzy part of the API. Go’s http pkg is a notorious one, with how a http.ResponseWriter can also be a Flusher and Hijacker and I don’t know what else. If you in your middleware want to wrap it, you need to implement those interfaces as well, or the functionality is lost. But they’re not part of the “visible” API, they’re type assertions buried deep in the standard library, good luck with that. For this reason Go 1.20 introduced the http.ResponseController.
- _yy01 2y ago
  I encountered this and went through some corners of the standard library. shameless plug to my own blog post: https://mahesh-hegde.github.io/posts/go-interface-smuggling/ https://mahesh-hegde.github.io/posts/go-interface-smuggling/
  - rollulus 2y ago
    If you never encountered this you never created your own ResponseWriter, or you did it and inadvertently broke its Flushing, Pushing and Hijacking capabilities. It’s not more complicated than this.
  - quectophoton 2y ago
    Wait what. For functionality like that[1] I would have expected them to fallback to `bufio.Peek` if the `fs.File` is not an `io.Seeker`. Sure, it's a bit slower, but it would just work. If someone implements custom `fs.File`, then it's up to them implementing any additional interface that allows better performance. It's the same idea as `io.Copy` trying to use first `io.WriterTo` or `io.ReaderFrom` if implemented, and then falling back to a manual copy. The thought process should be: First try to ask for an interface that has all methods you require (that's like the whole point of an interface). If you can't do that, then try to work with the methods you have, and then mention that performance can be improved if additional interfaces are implemented. Only if when you can't do that, you fail with a runtime error complaining about missing methods you didn't ask for in the interface signature. I might be missing some reason that prevents them from using `bufio.Peek` though. [1]: Link to the use of Seek, for reference: https://github.com/golang/go/lob/e8ee1dc4f9e2632ba1018610d1a1187743ae397f/src/net/http/fs.go#L289 https://github.com/golang/go/lob/e8ee1dc4f9e2632ba1018610d1a...
hmage 2y ago
November 5, 2014
- dang 2y ago
  Year added above. Thanks!
MrBuddyCasino 2y ago
Am I missing something or is this a lot of words to describe the equivalent of the JVMs instanceof / type casting operator?
- neonsunset 2y ago
  It is. What's with the languages and their communities trying to reinvent worse wheel and label it with a new word all the time? if (thing is IFasterThing ft) { ft.FastPath(arg1, arg2); }
  - blev 2y ago
    The author’s mistake was assuming that interfaces are unique to Go. I think it’s still a helpful article as someone who works frequently with interfaces because, while I “discovered” the interface upgrade idea on my own, I had never considered the proxy problem.
    - philosopher1234 2y ago
      Go’s interfaces are unique in that they are structural, which makes this problem more difficult to solve.
karmakaze 2y ago
This is basically a hack to get pragmatic performance at a cost of comprehensibility. It would have been better if Go allowed union types then a func could be declared to use interface A|B efficiently. Passing a narrow interface when a wider implemented one could get used is lying about the actual "interface" in the signature. Added to that is that Go's interfaces are structural (which I think are great), but could lead to accidental misuse by passing in interface A for an object that also has method X for unrelated purposes that co-incidentally satisfies interface B which the called func on A magically uses. > Like all articles about Go’s interfaces, we are obligated to start with Go’s io package. Also the io package, or stdlib in general is not a good place to look for good patterns to use in Go. Numerous antipatterns are used in the name of performance. The principles for stdlib authors and recommendation for Go developers are different. As an example io functions can return a value AND an error--and whether to continue or not depends on the specific error (as some are benign). It's better that I don't name an example as you should always be on the lookout for (until having learned) them.
- 38 2y ago
  > It's better that I don't name an example as you should always be on the lookout for (until having learned) them. "this gun will shoot you in the foot in some cases. It's better that I don't name an example as you should always be on the lookout for (until having shot yourself in the foot) them"
  - karmakaze 2y ago
    They are all documented in the io package. There's no point in telling you just one of them. But you have to RTM which was my point.
yawz 2y ago
The title needs "2014" to be added.
- dang 2y ago
  Added. Thanks!
dang 2y ago
Discussed at the time: Interface Upgrades in Go - https://news.ycombinator.com/item?id=8714051 https://news.ycombinator.com/item?id=8714051 - Dec 2014 (40 comments)
geoka9 2y ago
> While it just so happens that all the ResponseWriters that net/http give you implement (e.g.) CloseNotifier, there’s not really any way for you to know that without reading the source. Go editor tooling helps with this: gopls can do this nowadays (e.g. `lsp-find-implementation` in Emacs) and go oracle/guru may have already supported this back in 2014. It works both for finding implementations of an interface and finding interfaces implemented by a type.

88 points by panic 2y ago

While the article highlights interface upgrades, it overlooks potential downsides like increased complexity in debugging. Anyone experienced issues with this?

PUSH_AX 2y ago

I remember working in a team that used interfaces for a ton of things. Every single one of those things only ever had one concrete implementation, still to this day. A lot of those things were also purely for mocking in tests too. Today I don’t use them, unless I need it retrospectively. Which I find is rare. This is not a knock on interfaces as a language feature. Just inserting a random anecdote about pragmatism..

radicalbyte 2y ago
So you don't test the parts of your system which need mocks to be effectively tested? How do you avoid the maintenance nightmare that causes? I'm working on a Golang project where we've not yet introduced those kind of tests and, although it can be quicker to code initially, it's slower overall. The project relies heavily on other Open Source systems so the Golang code isn't even that complex. I can't imagine doing something hard and keeping it working over time without being to isolate dependencies to enable testing and keep the structure clear overall.
- PUSH_AX 2y ago
  I’m saying we did test parts that required mocks, and we used interfaces as a vehicle to streamline that. Rightly or wrongly.
  - PUSH_AX 2y ago
    I’ve just realised you understood and we’re asking present tense I think. The truth is I also do less unit testing, identifying instead where I know confidence is needed and where it’s not.
- the_gipsy 2y ago
  The question is, are those interfaces used for anything but mocking tests? If not, could there be a better solution?
  - valzam 2y ago
    I keep hearing this but I don't quite understand what the alternative is? I get that maybe introducing interfaces within your own business domain CAN be overused but certainly any external depedency should be fronted by an interface so the rest of the code can be tested without spinning up test http servers, database containers etc. all over the place. It just feels like a really lightweight, simnple thing to have an interface for some API that you are calling. It also means you are less depedent on the concrete implementation of the API and don't leak types and behaviour from systems you don't control into your codebase
    - afavour 2y ago
      This is a situation where I enjoy coding in TypeScript. You don’t need to front with interfaces, you just mock objects that map 1:1 with the expected API. So you write your code targeting the standard Fetch API then just pass in a mock when testing.
    - ashishmax31 2y ago
      +1 to this. Especially with go, most of the (non-integration) tests I end up writing are behaviour driven tests. I'm not really sure how one can write these sort of tests without interfaces and mocks.
      - randomdata 2y ago
        One could use build tags to provide alternative implementations over a concrete type, I suppose. But like with everything in life, there is no right solution. Just different tradeoffs. If interfaces best align with the trades you are willing to make, go for it.
    - dgb23 2y ago
      Instead of pushing the external, side-effecting thing down the stack, I put it up the stack and call the logic that generates the data for it from there. Now I don't need to mock an interface, I just need to test data in/out. If you do this consistently, then you have much of the side effecting code residing next to each other per process/input handler etc. It's not always feasible or the right thing. But it's a good default way of structuring code and makes testing much more straight forward. And it makes more clear "at a glance" what happens because you can reason more locally about the side effecting parts. > It also means you are less depedent on the concrete implementation of the API and don't leak types and behaviour from systems you don't control into your codebase You are always dependent on that. Whether you hide it down the stack or lift it up, you still need to put your data into a specific shape, do the same essential checks etc.
jen20 2y ago
Something one often sees in Go (which is in my mind an anti-pattern) is the supplier of an object also supplying an equivalent interface, as they would in Java or another nominative adoption system. In Go, the interface belongs to the consumer!
- slekker 2y ago
  But how do you avoid duplicating the interface in many places? We've tried both approaches, declaring the interface in one package and importing it where used but also declaring it directly colocated with the consumer. I've found the first approach worked better when we had to add or modify the interface methods
  - jen20 2y ago
    You don’t. Each consumer of something declares what it needs via an interface, something concrete may be passed that implements it - including fakes for tests. Of course this doesn’t apply when using interfaces for polymorphic return types in a library, but that is a distinct use case for interfaces to address. Perhaps some of this is just taste, but this is the way I’ve found interfaces in Go to be most useful.

nickcw 2y ago

I used to use interface upgrades in rclone (a program which syncs your data to many different cloud providers) for all the optional features a backend (cloud provider) might have, for example renaming a file. However the proxy problem became unmanageable. That's when I had a backend (eg the crypt backend which encrypts any other backend) which wraps another backend. Can the crypt backend rename things? That depends on what it is wrapping and you'd have to upgrade the interface call it and then get a special error to find out. Eventually I switched to a table of bound method pointers which were easy to test against nil to see whether they were implemented and could be filled with a very small amount of reflection. In my experience interface upgrades are useful but the proxy problem is very real so use sparingly only! I did suggest at one point (to rsc) an addition to Go which would allow interface proxies to take methods away from their method set at run time to fix this problem, so the proxy could only have the methods it could implement. I've no idea how difficult this would be to implement though.

lazypenguin 2y ago
Thanks for rclone, great tool!
memset 2y ago
Could you share a very small code snippet describing what you mean by bound method pointers? (Huge fan of rclone!)
- jerf 2y ago
  Something like this: https://go.dev/play/p/AVVz0qs2i1e https://go.dev/play/p/AVVz0qs2i1e This is much less convenient than using interfaces directly when that is possible, but comes with a lot more power. You have more insight into the "interface" value, you can use this to manually implement prototype-like behavior as seen in Javascript, you can deliberately make an object where one "method" is invoked on one object and one "method" is invoked on another, which is hard to read but may be just what you need in some situation, etc. There are a lot of ways to improvise on this tune. Also note most OO or OO-flavored languages can do this, Go has no special claim to it. But the tradeoffs are pretty similar in all those languages... you want to be sure this is what you need before you use it, because the design costs are significant.

rollulus 2y ago

Note that this is not necessarily a great thing to overuse. Also note that the article is 10 years old. Relying on such upgrades sort of introduces a dark and fuzzy part of the API. Go’s http pkg is a notorious one, with how a http.ResponseWriter can also be a Flusher and Hijacker and I don’t know what else. If you in your middleware want to wrap it, you need to implement those interfaces as well, or the functionality is lost. But they’re not part of the “visible” API, they’re type assertions buried deep in the standard library, good luck with that. For this reason Go 1.20 introduced the http.ResponseController.

_yy01 2y ago
I encountered this and went through some corners of the standard library. shameless plug to my own blog post: https://mahesh-hegde.github.io/posts/go-interface-smuggling/ https://mahesh-hegde.github.io/posts/go-interface-smuggling/
- rollulus 2y ago
  If you never encountered this you never created your own ResponseWriter, or you did it and inadvertently broke its Flushing, Pushing and Hijacking capabilities. It’s not more complicated than this.
- quectophoton 2y ago
  Wait what. For functionality like that[1] I would have expected them to fallback to `bufio.Peek` if the `fs.File` is not an `io.Seeker`. Sure, it's a bit slower, but it would just work. If someone implements custom `fs.File`, then it's up to them implementing any additional interface that allows better performance. It's the same idea as `io.Copy` trying to use first `io.WriterTo` or `io.ReaderFrom` if implemented, and then falling back to a manual copy. The thought process should be: First try to ask for an interface that has all methods you require (that's like the whole point of an interface). If you can't do that, then try to work with the methods you have, and then mention that performance can be improved if additional interfaces are implemented. Only if when you can't do that, you fail with a runtime error complaining about missing methods you didn't ask for in the interface signature. I might be missing some reason that prevents them from using `bufio.Peek` though. [1]: Link to the use of Seek, for reference: https://github.com/golang/go/lob/e8ee1dc4f9e2632ba1018610d1a1187743ae397f/src/net/http/fs.go#L289 https://github.com/golang/go/lob/e8ee1dc4f9e2632ba1018610d1a...

hmage 2y ago

November 5, 2014

dang 2y ago
Year added above. Thanks!

MrBuddyCasino 2y ago

Am I missing something or is this a lot of words to describe the equivalent of the JVMs instanceof / type casting operator?

neonsunset 2y ago
It is. What's with the languages and their communities trying to reinvent worse wheel and label it with a new word all the time? if (thing is IFasterThing ft) { ft.FastPath(arg1, arg2); }
- blev 2y ago
  The author’s mistake was assuming that interfaces are unique to Go. I think it’s still a helpful article as someone who works frequently with interfaces because, while I “discovered” the interface upgrade idea on my own, I had never considered the proxy problem.
  - philosopher1234 2y ago
    Go’s interfaces are unique in that they are structural, which makes this problem more difficult to solve.

karmakaze 2y ago

This is basically a hack to get pragmatic performance at a cost of comprehensibility. It would have been better if Go allowed union types then a func could be declared to use interface A|B efficiently. Passing a narrow interface when a wider implemented one could get used is lying about the actual "interface" in the signature. Added to that is that Go's interfaces are structural (which I think are great), but could lead to accidental misuse by passing in interface A for an object that also has method X for unrelated purposes that co-incidentally satisfies interface B which the called func on A magically uses. > Like all articles about Go’s interfaces, we are obligated to start with Go’s io package. Also the io package, or stdlib in general is not a good place to look for good patterns to use in Go. Numerous antipatterns are used in the name of performance. The principles for stdlib authors and recommendation for Go developers are different. As an example io functions can return a value AND an error--and whether to continue or not depends on the specific error (as some are benign). It's better that I don't name an example as you should always be on the lookout for (until having learned) them.

38 2y ago
> It's better that I don't name an example as you should always be on the lookout for (until having learned) them. "this gun will shoot you in the foot in some cases. It's better that I don't name an example as you should always be on the lookout for (until having shot yourself in the foot) them"
- karmakaze 2y ago
  They are all documented in the io package. There's no point in telling you just one of them. But you have to RTM which was my point.

yawz 2y ago

The title needs "2014" to be added.

dang 2y ago
Added. Thanks!

dang 2y ago

Discussed at the time: Interface Upgrades in Go - https://news.ycombinator.com/item?id=8714051 https://news.ycombinator.com/item?id=8714051 - Dec 2014 (40 comments)

geoka9 2y ago

> While it just so happens that all the ResponseWriters that net/http give you implement (e.g.) CloseNotifier, there’s not really any way for you to know that without reading the source. Go editor tooling helps with this: gopls can do this nowadays (e.g. `lsp-find-implementation` in Emacs) and go oracle/guru may have already supported this back in 2014. It works both for finding implementations of an interface and finding interfaces implemented by a type.