5 ms·

I guess it was good before compression. Nowdays working with protobufs is painful inside Google as well, but at least it's supported by everything. Most of wha

I guess it was good before compression. Nowdays working with protobufs is painful inside Google as well, but at least it's supported by everything. Most of what the CPUs at Google are doing is just copying fields from one protobuffer to another.

cogman10 6y ago
I think that's a fair description of pretty much every webapp :D Most of them are doing nothing more than copying data from the database into an http stream.
- ithkuil 6y ago
  Well, all what CPUs do is copying memory from one memory location to another, perform some arithmetic and do some conditional jumps :-)
  - cogman10 6y ago
    pop pop jump jump oh what a crunch it is. (sung to this tune https://www.youtube.com/watch?v=iENQXIQ8wH0 https://www.youtube.com/watch?v=iENQXIQ8wH0 ) Side note: It really is incredible what happened in the early days of computing when memory and computation were limited. How much care was taken in the precise layout of memory or even the timing of a calculation was insane.
  - xiphias2 6y ago
    Most of the arithmetic is checking optional fields if they are empty or not before copying :)
- xiphias2 6y ago
  Sure, but when you are using C++, protobufs are not the best way to store data...but I guess it could be worse.
- munk-a 6y ago
  Don't forget the validation! Webapps are dumb middleware that pipes data from the database into an http stream - but it needs to determine which database calls to invoke and sanitize all the incoming junk.
  - lmeyerov 6y ago
    We use arrow to do the validation 'for free': it has a typed columnar schema, so by passing the schema, we get the recordset validation without inspecting values. we still need to do semantic validation, but w/ arrow, now we do that in bulk and on gpus :)
philsnow 6y ago
At least when I was there, proto (de)serialization consumed the plurality of cpu cycles, but not the majority. It isn't really the majority these days, is it?

cogman10 6y ago

I think that's a fair description of pretty much every webapp :D Most of them are doing nothing more than copying data from the database into an http stream.

ithkuil 6y ago
Well, all what CPUs do is copying memory from one memory location to another, perform some arithmetic and do some conditional jumps :-)
- cogman10 6y ago
  pop pop jump jump oh what a crunch it is. (sung to this tune https://www.youtube.com/watch?v=iENQXIQ8wH0 https://www.youtube.com/watch?v=iENQXIQ8wH0 ) Side note: It really is incredible what happened in the early days of computing when memory and computation were limited. How much care was taken in the precise layout of memory or even the timing of a calculation was insane.
- xiphias2 6y ago
  Most of the arithmetic is checking optional fields if they are empty or not before copying :)
xiphias2 6y ago
Sure, but when you are using C++, protobufs are not the best way to store data...but I guess it could be worse.
munk-a 6y ago
Don't forget the validation! Webapps are dumb middleware that pipes data from the database into an http stream - but it needs to determine which database calls to invoke and sanitize all the incoming junk.
- lmeyerov 6y ago
  We use arrow to do the validation 'for free': it has a typed columnar schema, so by passing the schema, we get the recordset validation without inspecting values. we still need to do semantic validation, but w/ arrow, now we do that in bulk and on gpus :)

philsnow 6y ago

At least when I was there, proto (de)serialization consumed the plurality of cpu cycles, but not the majority. It isn't really the majority these days, is it?