3 ms·
Saddest thing is we're reading these values from a database, and the database SDK does not have a mechanism for reading values into user-controlled buffers with
by asdfasgasdgasdg 6y ago
Saddest thing is we're reading these values from a database, and the database SDK does not have a mechanism for reading values into user-controlled buffers without copying. That is a conceptually neat solution to the problem, but since it's a somewhat strange thing to want to do, we can't do it in our case. Storing the extended values is something I also considered, but there are tens or hundreds of billions of them in production. Couldn't really ask for an extra PB of space for the relatively small performance and elegance benefit you get from not copying these things.
- pedrocr 6y agoExtra copying is the downside if you don't control what you're reading from. Thankfully memcpy is fast but if your input is very big it's still painful. In rust I just used an API that allows reading from both an input stream and memory so that if you are reading directly from a file the copy is avoided: https://github.com/pedrocr/rawloader/blob/c793a132fa03e336f8c5c078f01d3ed1a3ee8c07/src/decoders/mod.rs#L107-L128 https://github.com/pedrocr/rawloader/blob/c793a132fa03e336f8...
- midjji 6y agoIs the extra copying even performed? It either has to be a small thing, or processed in chunks, or oom might be a problem. So its small, in this case, will the compiler really copy, or might it just optimize the operation away?
- pedrocr 6y agoIf you feed it an N byte array it will have to allocate an N+16 array and memcpy. If you're reading from the network/disk/etc then there is no need to copy if you can allocate the larger array directly.