3 ms·

Data format is absolutely important for operating system IPC. JSON is raw text. Messages occupy more RAM than a binary format, which increases your working set,

Data format is absolutely important for operating system IPC. JSON is raw text. Messages occupy more RAM than a binary format, which increases your working set, and it requires at least some nontrivial parsing. This ultimately limits the number of messages you can send per second.

jcelerier 2y ago
It's not that simple: [1.2,1.3,1.4] is 13 bytes, the equivalent binary storage is 3*sizeof(double) (24 bytes) + whatever overhead exists for representing a dynamic array in your language
- immibis 2y ago
  What's the chance the data you want to transmit is exactly 1.2 rather than 1.205338843375?
- gpderetta 2y ago
  If you expect to routinely send small numbers your serialization protocol can use floats or some small fix point format. Variable length binary number encodings are also an option. The dynamic array overhead in a language is immaterial when discussion serialization formats.
- jchw 2y ago
  Personal opinion: don't put floating point numbers into an API. If you really want real numbers passing them as decimal string is actually sensible. This can be done without using JSON, of course. Can also do fixed precision as well, which will store reasonably well into a VLQ. OTOH this is pretty cherry picked. It is questionable if you really need floating point numbers if your numbers are all this small and low precision, but in actuality unless this comprises your entire message, you'll probably still lose to JSON overhead eventually anyways, whereas non-self-describing serialization like capnp has effectively no overhead by default. The advantage of encoding as doubles is that it is predictable and won't explode into taking massively more, its always the same size per number. If you want it to be smaller for some reason, compression is always an option, though I suspect for IPC it's the wrong thing to be concerned about.
  - Netch 2y ago
    > If you really want real numbers passing them as decimal string is actually sensible. Hexadecimal, if so. Nearly any decent modern language and runtime can it. $ python >>> import math >>> math.pi.hex() '0x1.921fb54442d18p+1' This gives exact value without any rounding effects and conversion cost. (I donʼt count decimal floating-point as it is quite rare and its application is faint. In overwhelming most of cases, fixed point is better.)

jcelerier 2y ago

It's not that simple: [1.2,1.3,1.4] is 13 bytes, the equivalent binary storage is 3*sizeof(double) (24 bytes) + whatever overhead exists for representing a dynamic array in your language

immibis 2y ago
What's the chance the data you want to transmit is exactly 1.2 rather than 1.205338843375?
gpderetta 2y ago
If you expect to routinely send small numbers your serialization protocol can use floats or some small fix point format. Variable length binary number encodings are also an option. The dynamic array overhead in a language is immaterial when discussion serialization formats.
jchw 2y ago
Personal opinion: don't put floating point numbers into an API. If you really want real numbers passing them as decimal string is actually sensible. This can be done without using JSON, of course. Can also do fixed precision as well, which will store reasonably well into a VLQ. OTOH this is pretty cherry picked. It is questionable if you really need floating point numbers if your numbers are all this small and low precision, but in actuality unless this comprises your entire message, you'll probably still lose to JSON overhead eventually anyways, whereas non-self-describing serialization like capnp has effectively no overhead by default. The advantage of encoding as doubles is that it is predictable and won't explode into taking massively more, its always the same size per number. If you want it to be smaller for some reason, compression is always an option, though I suspect for IPC it's the wrong thing to be concerned about.
- Netch 2y ago
  > If you really want real numbers passing them as decimal string is actually sensible. Hexadecimal, if so. Nearly any decent modern language and runtime can it. $ python >>> import math >>> math.pi.hex() '0x1.921fb54442d18p+1' This gives exact value without any rounding effects and conversion cost. (I donʼt count decimal floating-point as it is quite rare and its application is faint. In overwhelming most of cases, fixed point is better.)