8 ms·

Go developer here. I posted this on the blog too. """ This is a mistake, plain and simple. It was pointed out on the Go developer mailing list a few days befor

by rsc 13y ago

Go developer here. I posted this on the blog too. """ This is a mistake, plain and simple. It was pointed out on the Go developer mailing list a few days before your post, and you can see my reply there. NewRequest should take an io.ReadCloser. Unfortunately, due to backwards compatibility, a long standing semantic mistake like this is not something we can just fix, but we've at least documented it. The tweet you quoted is wrong, and I want to explain why. If an interface value v is passed to a function f, then what does happen from time to time is that f will look for custom methods on v that are at least logically equivalent to the static interface type f is declared to expect. For example, io.Copy takes an io.Reader and an io.Writer and does the obvious thing, Reading into a buffer and then passing that buffer to Write. Of course, it would be nice to bypass the buffer when possible, and so there is a standard WriterTo interface that an io.Reader can also implement that, in contrast to Read, which copies into a buffer, says "read the data out of yourself and directly into this Writer". If io.Copy finds that method, it will use it, but the operation - reading - is the same. There is an analogous case for the Writer too. If you pass a value v to json.Marshal, Marshal will check to see if v implements json.Marshaler, meaning v knows how to marshal itself as JSON, in which case v is given that opportunity. Again, same operation you requested. In contrast, if you have an io.Reader, while it might be okay to look for other Read-like methods, it is certainly not okay to look for Close. That's a very different operation, it's surprising, and it shouldn't be there. FWIW, I'm not aware of any other instances of this kind of semantic mixup in the standard library. But again, unfortunately, we can't change it until Go 2. All we can do for now is document it and move on. """

marshray 13y ago
Out of simple idle language-design-bystander curiosity, can you recall any other times where this class of bug has bitten?
- rsc 13y ago
  No. A related bug would be if you implement Read and (say) ReadByte, but your ReadByte is buggy. Then when you get weird behavior passing yourself to something that expects a Reader (but internally needs byte-at-a-time and uses ReadByte to avoid double-buffering the input) it can take a minute or two to realize that ReadByte might be the problem and not Read. I've only seen that once or twice.
  - deleted 13y ago
    [deleted]
deleted 13y ago
[deleted]
manojlds 13y ago
I know of atleast one instance in C# where something like this happens. The IEnumberable<T>.Count() extension sees if the input is actually a ICollection and uses the count on that to get the count directly, rather than iterating over it. I remember writing extension methods in C# for IEnumerable<T>, which would of course take in an IEnumerable<T>, but saw if the actual input was a ICollection<T>, IList<T> etc to optimize the operations.
- benhoyt 13y ago
  I'm not a heavy C# dev, but I don't understand why your first paragraph would be so: isn't that the point of implementing an interface like IEnumerable<T> -- so that you can implement Count() in an optimized, specialized way that "gets the count directly"?
  - RexM 13y ago
    Some implementations of IEnumerable do have a Count property on them. He's specifically talking about the Count() extension method. Since this extension method applies to all IEnumerable<T> implementations, there may or may not be an optimized way of accessing the count, and this method would need to know about them. The code looks a little something like this (decompiled from the .NET framework, so the variable names are probably not true to what they're actually named in the real source) https://gist.github.com/anonymous/7c2c65268feafa66fb02 https://gist.github.com/anonymous/7c2c65268feafa66fb02
  - halter73 13y ago
    Count() is an extension method for IEnumerable<T>. This means that Count() is a static method which takes an single argument of type IEnumerable<T> and ostensibly only depends on members defined by IEnumerable<T> to work (but manojlds points out that this isn't really the case). C# provides syntactic sugar to make calling a static extension method look like calling an instance method. If you wanted to implement your own "Count" you would implement ICollection<T> which the IEnumerable<T> extension method would call into.
  - rat87 13y ago
    (I'm not a c# expert and I may be wrong) IEnumerable really only requires a IEnumerator GetEnumerator() method most of the real "meat" of ienumerable is in the static System.Linq.Enumerable class. You see c# doesn't have multiple inheritence or scala like traits or type classes but it does have a cool compiler hack called Extension methods(http://msdn.microsoft.com/en-us/library/vstudio/bb383977.aspx http://msdn.microsoft.com/en-us/library/vstudio/bb383977.asp...). You take a static class with static helper functions like you might make in java and you add a this keyword to the first argument then you can call it with a syntax that makes it look like it was a method on that class. For example something like public static class Enumerable { public static int Count<TSource>(this IEnumerable<TSource> source { int i=0; for(TSource t in source) { i++; } return i; ) } And if you use Extensions methods on an interface type you get all of the extension methods for free once you implement the minimal interface. One of the downsides is that you can only really define it in one place and can't override it and if you want even somewhat efficient type specialized versions you need to cast. I believe in scala you can use traits in a similar manner for default implementation and specialize them for collections but traits are more complicated/powerfull feature.
deleted 13y ago
[deleted]
stiff 13y ago
You adopted "looking for custom methods" without making it part of the contract in any way as a best practice in the standard library, but it is a "mistake, plain and simple"? Doesn't it, like, illustrate a serious language and/or standard library design issue? What can possibly motivate breaking encapsulation in the standard library? Why even have interfaces if even the standard library will abuse them? Your response is not very convincing...
- Locke1689 13y ago
  Theoretically, if the standard library only optimizes about interfaces whose semantics are in the standard library (i.e., standard library interfaces) that's perfectly fine. What worries me about this approach is that Go is essentially duck-typed -- implementing an interface isn't necessary for something to appear like it implements an interface, and a method signature doesn't implicitly carry its semantic information alongside it.
  - jefffoster 13y ago
    Something like the proposal for C++ Concepts captures some of the semantics via axioms (http://en.wikipedia.org/wiki/Concepts_(C%2B%2B)#Axioms http://en.wikipedia.org/wiki/Concepts_(C%2B%2B)#Axioms) seems like a really interesting solution for this idea.
- mostafah 13y ago
  It’s not like that. It makes sense in Go. Say you have a type, and you implement “Read” function for it. You don’t do anything else but now you are implementing an interface and other functions can call your “Read” function. Having a “ReadByte” function and other functions calling that isn’t that much different from your perspective. Any type that implements “Reader” may implement similar functions too. And if they do, that’s probably for a reason. If your type has a specific “ReadByte” function you want others to use that instead of the generic “Read” for reading a single byte.
  - Someone 13y ago
    Unless your ReadByte function does something different than what the function you call assumes it does. For example, it might have different behaviour in edge cases (end of file, I/O exception). One could call that bad architecture, but one could not even be aware of the existence of the broader interface that has a ReadByte member function, or one could (IMO rightfully) think the difference does not matter of you call a method taking an argument of interface type that does not include ReadByte. In the absolutely worst case, it does something completely unrelated to what Read does. Unlikely? Yes, but it might be part of a badly factored class that also has code for reading from an A/D converter.
epidemian 13y ago
> what does happen from time to time is that f will look for custom methods on v that are at least logically equivalent to the static interface type f is declared to expect. Interesting. This kinda sounds good in practice, assuming programmers respect the interface's semantics when downcasting. But from a language design point of view it strikes me as a bit hacky al least, and at odds with the "Tell, Don't Ask" principle. Couldn't this need for downcasting be avoided by allowing something like default method implementations in interfaces? I'm asking this without any practical knowledge of the Go language; i'm sure different approaches have been considered, but i'm interested in understanding the reasons behind the preference for downcasting :) Imagine the Reader interface declared, besides the Read method, a WriteTo method with a default implementation: type Reader interface { Read(p []byte) (n int, err error) WriteTo(w Writer) (n int64, err error) { // Default implementation in terms of Read and w.Write // using a buffer just like io.Copy } } Then any type with a Read([]byte) (int, err) could still be referenced as a Reader, and when seen as a Reader one could call the convenient WriteTo method without downcasting. io.Copy could then tell the objects what to do instead of asking them what they are: func Copy(dst Writer, src Reader) (written int64, err error) { return src.WriteTo(dst) } A type that complies with the Reader interface could also implement a specific version of WriteTo if needed for performance reasons. This way not only programmers could avoid resorting to downcasting, but the interfaces would become more explicit: instead Reader being "something with a Read method... and maybe a WriteTo if you want to, but this is not declared in Reader itself" it'd be "something with a Read method and a default WriteTo mehod". The downsides of this would be that the dispatch semantics become a bit more complex for having to consider the default implementations, and maybe the interfaces could become more "bloated" (though i think that's subjective, as the semantics of an interface like Reader are implicitly mixed up with things like WriterTo; having all the methods in Reader would make that existing coupling more explicit).

by rsc 13y ago

marshray 13y ago

Out of simple idle language-design-bystander curiosity, can you recall any other times where this class of bug has bitten?

rsc 13y ago
No. A related bug would be if you implement Read and (say) ReadByte, but your ReadByte is buggy. Then when you get weird behavior passing yourself to something that expects a Reader (but internally needs byte-at-a-time and uses ReadByte to avoid double-buffering the input) it can take a minute or two to realize that ReadByte might be the problem and not Read. I've only seen that once or twice.
- deleted 13y ago
  [deleted]

deleted 13y ago

[deleted]

manojlds 13y ago

I know of atleast one instance in C# where something like this happens. The IEnumberable<T>.Count() extension sees if the input is actually a ICollection and uses the count on that to get the count directly, rather than iterating over it. I remember writing extension methods in C# for IEnumerable<T>, which would of course take in an IEnumerable<T>, but saw if the actual input was a ICollection<T>, IList<T> etc to optimize the operations.

benhoyt 13y ago
I'm not a heavy C# dev, but I don't understand why your first paragraph would be so: isn't that the point of implementing an interface like IEnumerable<T> -- so that you can implement Count() in an optimized, specialized way that "gets the count directly"?
- RexM 13y ago
  Some implementations of IEnumerable do have a Count property on them. He's specifically talking about the Count() extension method. Since this extension method applies to all IEnumerable<T> implementations, there may or may not be an optimized way of accessing the count, and this method would need to know about them. The code looks a little something like this (decompiled from the .NET framework, so the variable names are probably not true to what they're actually named in the real source) https://gist.github.com/anonymous/7c2c65268feafa66fb02 https://gist.github.com/anonymous/7c2c65268feafa66fb02
- halter73 13y ago
  Count() is an extension method for IEnumerable<T>. This means that Count() is a static method which takes an single argument of type IEnumerable<T> and ostensibly only depends on members defined by IEnumerable<T> to work (but manojlds points out that this isn't really the case). C# provides syntactic sugar to make calling a static extension method look like calling an instance method. If you wanted to implement your own "Count" you would implement ICollection<T> which the IEnumerable<T> extension method would call into.
- rat87 13y ago
  (I'm not a c# expert and I may be wrong) IEnumerable really only requires a IEnumerator GetEnumerator() method most of the real "meat" of ienumerable is in the static System.Linq.Enumerable class. You see c# doesn't have multiple inheritence or scala like traits or type classes but it does have a cool compiler hack called Extension methods(http://msdn.microsoft.com/en-us/library/vstudio/bb383977.aspx http://msdn.microsoft.com/en-us/library/vstudio/bb383977.asp...). You take a static class with static helper functions like you might make in java and you add a this keyword to the first argument then you can call it with a syntax that makes it look like it was a method on that class. For example something like public static class Enumerable { public static int Count<TSource>(this IEnumerable<TSource> source { int i=0; for(TSource t in source) { i++; } return i; ) } And if you use Extensions methods on an interface type you get all of the extension methods for free once you implement the minimal interface. One of the downsides is that you can only really define it in one place and can't override it and if you want even somewhat efficient type specialized versions you need to cast. I believe in scala you can use traits in a similar manner for default implementation and specialize them for collections but traits are more complicated/powerfull feature.

deleted 13y ago

[deleted]

stiff 13y ago

You adopted "looking for custom methods" without making it part of the contract in any way as a best practice in the standard library, but it is a "mistake, plain and simple"? Doesn't it, like, illustrate a serious language and/or standard library design issue? What can possibly motivate breaking encapsulation in the standard library? Why even have interfaces if even the standard library will abuse them? Your response is not very convincing...

Locke1689 13y ago
Theoretically, if the standard library only optimizes about interfaces whose semantics are in the standard library (i.e., standard library interfaces) that's perfectly fine. What worries me about this approach is that Go is essentially duck-typed -- implementing an interface isn't necessary for something to appear like it implements an interface, and a method signature doesn't implicitly carry its semantic information alongside it.
- jefffoster 13y ago
  Something like the proposal for C++ Concepts captures some of the semantics via axioms (http://en.wikipedia.org/wiki/Concepts_(C%2B%2B)#Axioms http://en.wikipedia.org/wiki/Concepts_(C%2B%2B)#Axioms) seems like a really interesting solution for this idea.
mostafah 13y ago
It’s not like that. It makes sense in Go. Say you have a type, and you implement “Read” function for it. You don’t do anything else but now you are implementing an interface and other functions can call your “Read” function. Having a “ReadByte” function and other functions calling that isn’t that much different from your perspective. Any type that implements “Reader” may implement similar functions too. And if they do, that’s probably for a reason. If your type has a specific “ReadByte” function you want others to use that instead of the generic “Read” for reading a single byte.
- Someone 13y ago
  Unless your ReadByte function does something different than what the function you call assumes it does. For example, it might have different behaviour in edge cases (end of file, I/O exception). One could call that bad architecture, but one could not even be aware of the existence of the broader interface that has a ReadByte member function, or one could (IMO rightfully) think the difference does not matter of you call a method taking an argument of interface type that does not include ReadByte. In the absolutely worst case, it does something completely unrelated to what Read does. Unlikely? Yes, but it might be part of a badly factored class that also has code for reading from an A/D converter.

epidemian 13y ago

> what does happen from time to time is that f will look for custom methods on v that are at least logically equivalent to the static interface type f is declared to expect. Interesting. This kinda sounds good in practice, assuming programmers respect the interface's semantics when downcasting. But from a language design point of view it strikes me as a bit hacky al least, and at odds with the "Tell, Don't Ask" principle. Couldn't this need for downcasting be avoided by allowing something like default method implementations in interfaces? I'm asking this without any practical knowledge of the Go language; i'm sure different approaches have been considered, but i'm interested in understanding the reasons behind the preference for downcasting :) Imagine the Reader interface declared, besides the Read method, a WriteTo method with a default implementation: type Reader interface { Read(p []byte) (n int, err error) WriteTo(w Writer) (n int64, err error) { // Default implementation in terms of Read and w.Write // using a buffer just like io.Copy } } Then any type with a Read([]byte) (int, err) could still be referenced as a Reader, and when seen as a Reader one could call the convenient WriteTo method without downcasting. io.Copy could then tell the objects what to do instead of asking them what they are: func Copy(dst Writer, src Reader) (written int64, err error) { return src.WriteTo(dst) } A type that complies with the Reader interface could also implement a specific version of WriteTo if needed for performance reasons. This way not only programmers could avoid resorting to downcasting, but the interfaces would become more explicit: instead Reader being "something with a Read method... and maybe a WriteTo if you want to, but this is not declared in Reader itself" it'd be "something with a Read method and a default WriteTo mehod". The downsides of this would be that the dispatch semantics become a bit more complex for having to consider the default implementations, and maybe the interfaces could become more "bloated" (though i think that's subjective, as the semantics of an interface like Reader are implicitly mixed up with things like WriterTo; having all the methods in Reader would make that existing coupling more explicit).