8 ms·

Interesting, I don't know Go (yet) but I was messing with it the other weekend. Can someone please summarize this []byte vs somethingReader thing for me? Assum

Interesting, I don't know Go (yet) but I was messing with it the other weekend. Can someone please summarize this []byte vs somethingReader thing for me? Assume I can program, just not familiar with Go. I was reading off sockets and it looked to me that the example code (i randomly ran into) had too many Reader something or other. Edit: Ok, I know what a streaming class does, they're available in many frameworks. I'm more interested in why you'd get forced to use them in the context of the Go standard library. Are they mandatory for sockets? Or for interacting with other common functions that I'd use to process the data out of my sockets? I just wanted to read up to a new line or a hard coded size limit from a socket... ;) Without getting accidentally quadratic in either cpu use or memory use...

mholt 1y ago
[]byte is a buffer. (io.)Reader is any type that can read bytes, like as part of a stream. There are `bytes.Reader` and `bufio.Reader` and `strings.Reader`, and many others, depending on what you're reading. These three types allow you to read []byte, any buffered input, and string, respectively, as streams. Streaming is more efficient for large pieces of data (files, etc; whatever you have), but the buffer is usually easier to work with and grants more flexibility.
ashishb 1y ago
Imagine you are reading files which can be of varying size (KB to GB). A byte array will do a contiguous allocation of the file size. A Reader can be much more thoughtful. And I say "can be" because someone can make Reader as inefficient as a byte array. Or they can read in chunks. For example, if you are trying you read exif data or reading up to first N bytes, Reader is a superior approach.
masklinn 1y ago
byte[] is a bytes buffer, a sequence of bytes. io.Reader is an abstraction around a read stream. You can adapt a byte[] to a Reader by wrapping it in a bytes.Reader, that way if a function needs a reader and you have bytes, you can give them your bytes. The problem TFA has, is that bytes.Reader implies a copy: it's going to read data into a second internal slice. So when all their library needs is the bytes, they could use the bytes themselves, and avoid a potentially expensive copy. Obviously you could just have a second entry point which takes straight byte[] instead of a reader, but in this case they're trying to conform to the standard library's image module[1] which does not expose a bytes interface and conditionally adds further indirection layers. [1]: https://pkg.go.dev/image https://pkg.go.dev/image
- nottorp 1y ago
  > is that bytes.Reader implies a copy Yes, that's what "smelled" wrong to me too. Is it avoidable or does Go's stdlib force you to go through xxx.Reader reading from yyy.Reader that is made from a zzz.Reader and a ttt.Reader?
  - masklinn 1y ago
    Unless you know the concrete reader types and they expose their underlying reader or concrete type, short of unsafe yes. And as the essay points out, neither bytes.Reader nor bufio.Reader expose the underlying type.
  - mholt 1y ago
    There's io.ReaderFrom and io.WriterTo, which can avoid internal buffering.
konart 1y ago
Go's source code (comment actually) says it all tbh: https://github.com/golang/go/blob/master/src/io/io.go#L55 https://github.com/golang/go/blob/master/src/io/io.go#L55 (very much) tldr: anything that implements `Read(p []byte) (n int, err error)` - is a Reader.

mholt 1y ago

[]byte is a buffer. (io.)Reader is any type that can read bytes, like as part of a stream. There are `bytes.Reader` and `bufio.Reader` and `strings.Reader`, and many others, depending on what you're reading. These three types allow you to read []byte, any buffered input, and string, respectively, as streams. Streaming is more efficient for large pieces of data (files, etc; whatever you have), but the buffer is usually easier to work with and grants more flexibility.

ashishb 1y ago

Imagine you are reading files which can be of varying size (KB to GB). A byte array will do a contiguous allocation of the file size. A Reader can be much more thoughtful. And I say "can be" because someone can make Reader as inefficient as a byte array. Or they can read in chunks. For example, if you are trying you read exif data or reading up to first N bytes, Reader is a superior approach.

masklinn 1y ago

byte[] is a bytes buffer, a sequence of bytes. io.Reader is an abstraction around a read stream. You can adapt a byte[] to a Reader by wrapping it in a bytes.Reader, that way if a function needs a reader and you have bytes, you can give them your bytes. The problem TFA has, is that bytes.Reader implies a copy: it's going to read data into a second internal slice. So when all their library needs is the bytes, they could use the bytes themselves, and avoid a potentially expensive copy. Obviously you could just have a second entry point which takes straight byte[] instead of a reader, but in this case they're trying to conform to the standard library's image module[1] which does not expose a bytes interface and conditionally adds further indirection layers. [1]: https://pkg.go.dev/image https://pkg.go.dev/image

nottorp 1y ago
> is that bytes.Reader implies a copy Yes, that's what "smelled" wrong to me too. Is it avoidable or does Go's stdlib force you to go through xxx.Reader reading from yyy.Reader that is made from a zzz.Reader and a ttt.Reader?
- masklinn 1y ago
  Unless you know the concrete reader types and they expose their underlying reader or concrete type, short of unsafe yes. And as the essay points out, neither bytes.Reader nor bufio.Reader expose the underlying type.
- mholt 1y ago
  There's io.ReaderFrom and io.WriterTo, which can avoid internal buffering.

konart 1y ago

Go's source code (comment actually) says it all tbh: https://github.com/golang/go/blob/master/src/io/io.go#L55 https://github.com/golang/go/blob/master/src/io/io.go#L55 (very much) tldr: anything that implements `Read(p []byte) (n int, err error)` - is a Reader.