3 ms·
fleet-wide profiling revealed that 25-35% of all CPU time was spent just moving bytes around[...]If data movement were faster, more work could be done on the sa
by userbinator 4y ago
fleet-wide profiling revealed that 25-35% of all CPU time was spent just moving bytes around[...]If data movement were faster, more work could be done on the same processors.
Here's a different thought experiment: how much of that data movement is actually necessary? If data movement was taking up so much time, why not investigate why and if that's even necessary, instead of thinking it's the hardware that needs to be faster? As the old saying goes, "the fastest way to do something is to not do it at all." One style of code that seems to be particularly exacerbated by the widespread use of HLLs is making lots of unnecessary temporary copies of data. The biggest inefficiency isn't the hardware (unless your goal is to sell more hardware, but that's a different rant...) --- it's the insanely complex and bloated layers of software that gets layered on top of what is otherwise perfectly adequate hardware. From that perspective, faster hardware only leads to even more wasteful and inefficient software. Making memory copies faster will only cause software to make more unnecessary copies.
But after reading the whole thing, I can say that this is certainly a very strange article. It's written by someone who is presumably quite knowledgeable about hardware, yet seems completely oblivious to the fact that x86 has had a dedicated instruction for this since the 8086, and 16 bytes per cycle was already achievable with the P6 (1996) "fast strings" enhancement:
https://stackoverflow.com/questions/33902068/what-setup-does-rep-do https://stackoverflow.com/questions/33902068/what-setup-does...
- hinkley 4y agoCynical answer: because that's not sexy. I have some coworkers trying to re-architect a big chunk of our application to selectively provide data because "only about 10% is used on each interaction". Nobody has bothered to check what % of the data is never used at all. If it's 50%, then you're complicating a whole lot of code and reuse semantics in order to get a 2-4x improvement in 'performance' that you're selling as a 10x improvement. But to your bigger question, defensive copies are about keeping someone else from munging your data. Borrow Semantics prevent some of those scenarios, which at least moves some of that inefficiency to compile time, where the tax is paid by the perpetrator and their peers, instead of by the users.
- maria2 4y agoMemory gets moved for a lot of reasons. Any type of FFI usually requires memory copies. For instance, if you pass a C string from Swift to C, the C side needs to copy the memory to use it after the immediate call. Or if C returns a string to Swift, Swift needs to copy the string into its own format.
- georgelyon 4y agoWhile the overall point is valid, this isn’t strictly true. A Swift string is frequently backed by a contiguous UTC8 buffer terminated with a \0, so passing to C is just a matter of passing the pointer and requires no copies.
- fathyb 4y agoThere is also the I/O, most interaction with disks or NICs are copying twice. We know efficient zero-copy I/O is achievable, it's just that it requires a lot of work for kernel and app developers with a very limited ROI.
- _dhruva 4y agoThis is my standpoint too. Assuming Google uses gRPC and protobufs extensively, I can only imagine a chaotic polyglot environment where buffers are copied and transformed very often. I work at a fairly large Python shop using libraries in C. Cost of FFI is non-trivial, memory management and moving data is the usual culprit.
- 4y ago
- titzer 4y agoA lot of the additional data movement is a.) defensive copies because too many data structures are mutable and b.) change of data format to internal software representations, e.g. wire->decode->IR1->...->IR1->encode->wire.
- robotresearcher 4y ago> change of data format to internal software representations There is so much of this. Is there any hope for a world where we can ship working memory without costly XDR? Does anyone have data on the rate of change of time or energy spent on XDR translation?
- titzer 4y agoI have thought a lot about this over the years, and I think one of the issues is that it is hard to specify fixed data formats in most programming languages (in a declarative way), leading to lots of code that decodes byte-by-byte or word-by-word. Since writing all that sucks, many programmers have turned to higher-level data formats such as JSON or protobufs that are decoded by libraries. I have been working on a new feature in Virgil to make writing explicit byte-by-byte layouts easier and directly supported by the compiler. It's still a work in progress though.
- nerpderp82 4y agoIf the heap were treated like a typed database, with transaction semantics, many of our problems with disappear. Layout should not be an implementation detail. I know Wasm won't solve this explicitly, but at least have a chance while everything is being redefined.
- robotresearcher 4y agoWouldn’t that move the data transformation costs from at I/O time to all the time?
- 4y ago
- ardel95 4y agoWhile you are right, we are where we are and there is tremendous leverage in making an extremely common (even if undesirable) thing more efficient. It kind of doesn't matter why millions of lines of code perform memcpys - they do, and we're unlikely to change majority of them anytime soon. P.S. I work on the type of systems where we obsess over unneeded memory accesses and copies. Minimizing those is not as trivial as it may seem, especially if you care about memory safety.
- vanderZwan 4y ago> While you are right, we are where we are and there is tremendous leverage in making an extremely common (even if undesirable) thing more efficient. Aren't you worried that this will just lead to another manifestation of the Jevons Paradox? I get that you already mention that everyone in your "field" understands that memory is the bottleneck to avoid, but still. [0] https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox
- CalChris 4y agoIt's a lot of work to reason about programs in order to reduce copies. A different, cheaper and more feasible approach is to reasoning about alignments and sizes.
- ilyt 4y agoI'd imagine a lot of that was just same simple operation on a steam of data; there isn't that much to optimize there aside from finding a way to operate on more data at a time
- touisteur 4y agoI've seen some long running work on the rust compiler to use high level guarantees (lifetimes, ownership...) to improve codegen and one of the big tasks seems to be 'remove unnecessary memory copies'. Fun to watch, I hope there'll be a cool write up.