4 ms·
I'm not sure I'd call varints a stunningly bad decision. It seems more like the kind of tradeoff one would make if storage and network transfer costs are consid
by jfim 6y ago
I'm not sure I'd call varints a stunningly bad decision. It seems more like the kind of tradeoff one would make if storage and network transfer costs are considered to be more important than the serialization/deserialization speed.
That being said, the fact that that particular tradeoff was considered to be good for Google doesn't mean it actually is, or that it's applicable to one's application.
- xiphias2 6y agoI guess it was good before compression. Nowdays working with protobufs is painful inside Google as well, but at least it's supported by everything. Most of what the CPUs at Google are doing is just copying fields from one protobuffer to another.
- cogman10 6y agoI think that's a fair description of pretty much every webapp :D Most of them are doing nothing more than copying data from the database into an http stream.
- ithkuil 6y agoWell, all what CPUs do is copying memory from one memory location to another, perform some arithmetic and do some conditional jumps :-)
- cogman10 6y agopop pop jump jump oh what a crunch it is. (sung to this tune https://www.youtube.com/watch?v=iENQXIQ8wH0 https://www.youtube.com/watch?v=iENQXIQ8wH0 ) Side note: It really is incredible what happened in the early days of computing when memory and computation were limited. How much care was taken in the precise layout of memory or even the timing of a calculation was insane.
- xiphias2 6y agoMost of the arithmetic is checking optional fields if they are empty or not before copying :)
- xiphias2 6y agoSure, but when you are using C++, protobufs are not the best way to store data...but I guess it could be worse.
- munk-a 6y agoDon't forget the validation! Webapps are dumb middleware that pipes data from the database into an http stream - but it needs to determine which database calls to invoke and sanitize all the incoming junk.
- lmeyerov 6y agoWe use arrow to do the validation 'for free': it has a typed columnar schema, so by passing the schema, we get the recordset validation without inspecting values. we still need to do semantic validation, but w/ arrow, now we do that in bulk and on gpus :)
- philsnow 6y agoAt least when I was there, proto (de)serialization consumed the plurality of cpu cycles, but not the majority. It isn't really the majority these days, is it?
- habitue 6y agoYou're right, it's probably fair to say it's a stunningly bad tradeoff for most applications most of the time, given we have fast compression like snappy & brotli available now and cloud costs are heavily weighted towards CPU costing more than storage & network transfer