4 ms·

Sorry to be pedantic, but there’s nothing “zero copy” in a computer. Every single operation copies a bunch of things around. Performance in computing is about

Sorry to be pedantic, but there’s nothing “zero copy” in a computer. Every single operation copies a bunch of things around. Performance in computing is about managing the copy granularity by trying to align cohesiveness of space and time (optimal batching) which also relies on end-to-end analysis not just one step in the computation (whether a specific operation is a copy, virtual copy /cow/, move or virtual move by reference etc.). Avoiding a copy has time costs, such as indirection and incidental coupling. You trade space for time. Sometimes that’s smart sometimes not. We have countless stories about how removing poor caching and going brute force speeds up an app. Specifically for serialization and storage we also have increasingly more examples of how storing something compressed and calculating it on demand from compressed storage is faster than avoiding a copy on it because of how heterogenous memory bandwidth to compute bandwidth is over time (i.e. slow IO fast compute). For example it’s often better to pack data into varints or compress sequences by delta rather than naively store them inRAM as long as they’re read roughly in sequence or groups. Avoiding a copy and transform feels nice, but often isn’t. It should be only applied when specific uses show compute is the bottleneck. Premature optimization yadda yadda.

ithkuil 3y ago
All you said is true and useful. Just to chime in on a detail: often the "zero-copy" term is used in the context of data "forwarding" use cases (serving data loaded from disks or passed through a proxy etc). The idea there is to have zero "extra" copies of the data. Obviously if you're reading data from disk and sending it over the network there is at least one copy going on: the copy from the disk to the network card buffer. And unless your network controller buffer is directly addressable by the disk controller, there is also going to be another copy of the data sitting in RAM. That's often still considered zero copy IO mainly because the CPU is not involved in the copying (which happens via DMA). In case of network to network forwarding the data can even stay within the same buffers on the same network controller. Protocols that involve sending large blobs of uninterpreted bytes are particularly amendable to this kind of optimization. For example you can send a large blob of data from disk and pump it on a TCP connection without having that data going through the CPU at all. Practical necessities as encryption make that even more challenging but there are some network controllers supporting hardware encryption so at least in principle that's still possible. However when you also throw data serialization in the mix you often end up getting the CPU involved in converting data from the raw form you read on disk and the encoded/serialized form in the network response. For example consider a server that returns a stream of data using the gRPC ByteStream API which is a gRPC stream of ReadResponse messages that contain a single `bytes` field. The protobuf encoding of that field is just a field tag plus the field size plus the raw bytes of the payload. In principle it could be possible to implement a gRPC serializer that avoid materializing the body of that data field in memory but instead rely on sendfile or other primitives that potentially can apply further optimizations and at least remove the CPU out of the loop. But doing that is hard, and affects the API of the serializer and every library that uses it. I'm not aware of any opensource protobuf+grpc implementations that do this (I may be wrong, long time since I checked and I may be missing something). I don't mind that the rest of the serialized message would need to be unpacked/repacked to/from different representations, as generally the size of that part is dwarfed by the large blobs.
- jeffbee 3y ago
  > In principle it could be possible to implement a gRPC serializer FWIW it isn't that much work to rig up a C++ gRPC implementation that just does whatever you want with some buffers. That is, you provide SerializationTraits for your type, and you provide the function that converts your type to what is the functional equivalent of an iovec. This helps minimize copies because, as in your example, it's often useful to add a preamble to some large byte buffer. Doing so in a naive codelab style with the vanilla protobuf API would result in extra copies, probably. However I also note that recent releases of protobuf C++ allow bytes fields to be represented by an absl::Cord, which enabled you to refer to existing large buffers instead of copying them. In either case you need a pointer that can be sent with `sendmsg`, not `sendfile`
  - ithkuil 3y ago
    Yeah I was talking about an API that would expose the file descriptors so you could use sendfile.
deleted 3y ago
[deleted]
notso411 3y ago
This fetishisation of zero copy, whatever that means, usually ends up in a less than optimal solution anyway.

ithkuil 3y ago

All you said is true and useful. Just to chime in on a detail: often the "zero-copy" term is used in the context of data "forwarding" use cases (serving data loaded from disks or passed through a proxy etc). The idea there is to have zero "extra" copies of the data. Obviously if you're reading data from disk and sending it over the network there is at least one copy going on: the copy from the disk to the network card buffer. And unless your network controller buffer is directly addressable by the disk controller, there is also going to be another copy of the data sitting in RAM. That's often still considered zero copy IO mainly because the CPU is not involved in the copying (which happens via DMA). In case of network to network forwarding the data can even stay within the same buffers on the same network controller. Protocols that involve sending large blobs of uninterpreted bytes are particularly amendable to this kind of optimization. For example you can send a large blob of data from disk and pump it on a TCP connection without having that data going through the CPU at all. Practical necessities as encryption make that even more challenging but there are some network controllers supporting hardware encryption so at least in principle that's still possible. However when you also throw data serialization in the mix you often end up getting the CPU involved in converting data from the raw form you read on disk and the encoded/serialized form in the network response. For example consider a server that returns a stream of data using the gRPC ByteStream API which is a gRPC stream of ReadResponse messages that contain a single `bytes` field. The protobuf encoding of that field is just a field tag plus the field size plus the raw bytes of the payload. In principle it could be possible to implement a gRPC serializer that avoid materializing the body of that data field in memory but instead rely on sendfile or other primitives that potentially can apply further optimizations and at least remove the CPU out of the loop. But doing that is hard, and affects the API of the serializer and every library that uses it. I'm not aware of any opensource protobuf+grpc implementations that do this (I may be wrong, long time since I checked and I may be missing something). I don't mind that the rest of the serialized message would need to be unpacked/repacked to/from different representations, as generally the size of that part is dwarfed by the large blobs.

jeffbee 3y ago
> In principle it could be possible to implement a gRPC serializer FWIW it isn't that much work to rig up a C++ gRPC implementation that just does whatever you want with some buffers. That is, you provide SerializationTraits for your type, and you provide the function that converts your type to what is the functional equivalent of an iovec. This helps minimize copies because, as in your example, it's often useful to add a preamble to some large byte buffer. Doing so in a naive codelab style with the vanilla protobuf API would result in extra copies, probably. However I also note that recent releases of protobuf C++ allow bytes fields to be represented by an absl::Cord, which enabled you to refer to existing large buffers instead of copying them. In either case you need a pointer that can be sent with `sendmsg`, not `sendfile`
- ithkuil 3y ago
  Yeah I was talking about an API that would expose the file descriptors so you could use sendfile.

deleted 3y ago

[deleted]

notso411 3y ago

This fetishisation of zero copy, whatever that means, usually ends up in a less than optimal solution anyway.