4 ms·

Serialization/deserialization is still quite costly even with protobufs/flat buffers. This is not the last tiny bit. In some applications, I've seen it take mor

• pclmulqdq 4y ago
  In some applications, this is true. I think 15% is pretty extreme, though, unless what you're doing is something like parsing/ creating data feeds (where the serialization is the point). In those cases, it's probably a good idea to have your own format. Still, JSON would be a lot worse for these cases than protobufs or flatbuffers. Also, are you sure you aren't compressing them if you're using 15% CPU?
  • bluepizza 4y ago
    Yeah, you almost got it :) It's a simple backing store for a few jobs that create data feeds. Agreed that JSON would be much worse. We have a hard requirement that all services must be Java based, and protobuf outperforms Gson and Jackson by miles. I have been trying to find the cheapest serialization/deserialization I could find for cache purposes. So far, the best option is a Guava/Caffeine cache, because we skip the serialization completely in this scenario, but so much more costly than having a good external cache.
    • pclmulqdq 4y ago
      If you are willing to do some DIY-ing, using finance-like encodings may be for you: See "Simple Binary Encoding" (https://www.fixtrading.org/standards/sbe/ https://www.fixtrading.org/standards/sbe/). The encoding scheme basically just uses structs with version numbering and IDs. There may be some Java code out there for SBE if you want to pull it off-the-shelf.
      • bluepizza 4y ago
        Fantastic! Thank you, this is really useful!