4 ms·

As a curiosity, a Linux iovec seems to have the same layout as an array of Go strings (a bunch of (pointer, length) pairs). Looks like it'd be straightforward t

As a curiosity, a Linux iovec seems to have the same layout as an array of Go strings (a bunch of (pointer, length) pairs). Looks like it'd be straightforward to implement a `func WriteStrings` taking a pointer to os.File and a []string that, on Linux, only makes one syscall. Or you could make your own struct of byte pointers and lengths to make an iovec you can fill with content from []byte's, with the caveat that the caller needs to leave those bytes untouched between when it adds them to the vec and the actual writev. To mix byte and string output, you probably need to use unsafe. I don't see a way to tie writev in with the stdlib text/template libraries, which stick to plain io.Reader and Writer, without forking them. Curious how writev compares to the usual approach of just pointing text/template at a *bufio.Writer (which also passes the "generate 40GB of the word DOOM and don't crash" test). Regardless of the specifics, BEAM and the ecosystem around it is one of the more interesting places to crib ideas from, in that it's pretty different from a lot of stuff out there but has shown some value and longevity in production.

jerf 11y ago
It would not be safe to attempt to provide a Go io.Writer that backs to some file descriptor and backs to writev. The text of the io.Writer interface forbids the io.Writer from modifying the passed-in []byte, which implies strongly that the caller is still considered to own it. Also, the fact that I said "implied" is awfully scary too. You can't buffer things easily because you can't reliably know when the last .Write is called, and all the solutions I can come up with will annihilate any conceivable performance advantage (i.e., if you're communicating across to some other goroutine, you just lost the performance). Especially since if you don't write it right away, you almost certainly must take a copy to be safe, and, oops, there went all the performance advantage again. The need to take a copy also prevents you from just declaring it has to be flushable; it's legal for something using a Writer to mutate a buffer and pass chunks of it in multiple times, and I'm pretty sure I've already got code that does that. No point in buffering up several slices if the caller is going to mutate it between invoking .Write calls. What you could do is create an object that implements, say, interface{ Multiwrite([][]byte) (int, error) }, that also implements io.Writer as a fallback in terms of the Multiwrite so this object is generally useful, but that still gets you no performance win for text/template unless it is modified to incorporate this new interface. However, that does seem like it's at least a possibility; there are, for instance, already optimizations in net/http for serving up files via the Linux sendfile kernel call, and this would be pretty similar. Providing implementations of Multiwriter for a file and a socket might not be a tough sell; complexifying text/template to actually use it in the core library might be, though. Depends on what kind of performance numbers you can show.
- twotwotwo 11y ago
  Yeah, writev seems different enough from write to merit a different Go interface. Hence the hypothetical WriteStrings method, for example, instead of building around Writer. Whether it'd be worth futzing to get templating with less copying depends on whether you can measure much practical benefit over bytes.Buffer, I guess. Takes round tuits to see. Mostly just thought writev was neat; it seems cool that it's theoretically possible to write Go code using it too.

jerf 11y ago

It would not be safe to attempt to provide a Go io.Writer that backs to some file descriptor and backs to writev. The text of the io.Writer interface forbids the io.Writer from modifying the passed-in []byte, which implies strongly that the caller is still considered to own it. Also, the fact that I said "implied" is awfully scary too. You can't buffer things easily because you can't reliably know when the last .Write is called, and all the solutions I can come up with will annihilate any conceivable performance advantage (i.e., if you're communicating across to some other goroutine, you just lost the performance). Especially since if you don't write it right away, you almost certainly must take a copy to be safe, and, oops, there went all the performance advantage again. The need to take a copy also prevents you from just declaring it has to be flushable; it's legal for something using a Writer to mutate a buffer and pass chunks of it in multiple times, and I'm pretty sure I've already got code that does that. No point in buffering up several slices if the caller is going to mutate it between invoking .Write calls. What you could do is create an object that implements, say, interface{ Multiwrite([][]byte) (int, error) }, that also implements io.Writer as a fallback in terms of the Multiwrite so this object is generally useful, but that still gets you no performance win for text/template unless it is modified to incorporate this new interface. However, that does seem like it's at least a possibility; there are, for instance, already optimizations in net/http for serving up files via the Linux sendfile kernel call, and this would be pretty similar. Providing implementations of Multiwriter for a file and a socket might not be a tough sell; complexifying text/template to actually use it in the core library might be, though. Depends on what kind of performance numbers you can show.

twotwotwo 11y ago
Yeah, writev seems different enough from write to merit a different Go interface. Hence the hypothetical WriteStrings method, for example, instead of building around Writer. Whether it'd be worth futzing to get templating with less copying depends on whether you can measure much practical benefit over bytes.Buffer, I guess. Takes round tuits to see. Mostly just thought writev was neat; it seems cool that it's theoretically possible to write Go code using it too.