5 ms·

Plaintext protocols are textbook case of a negative externality. Programmers who work on them capture value, but impose higher costs (vs a well-suited binary pr

Plaintext protocols are textbook case of a negative externality. Programmers who work on them capture value, but impose higher costs (vs a well-suited binary protocol) for parsing, serialising, transmission, storage, computation etc onto everyone else.

andrewflnr 6y ago
Interesting perspective, but considering how much everyone seems to want software, I think we have to say that the commons also captures a lot of the same value from plain text that the programmers do. That might work out to less of a negative externality than just a trade-off for everyone, especially when you consider the positive effects on the commons of the network effects that text makes easier.
- jacques_chester 6y ago
  Interesting counterpoint. It would come down to how things net out. And it won't be stable over time. I'm still of the view that programmers systematically overvalue their convenience, because it's a value/cost tradeoff that they directly experience.
  - andrewflnr 6y ago
    Fair. I do think the long term goal should be a compact binary format with equal or better tooling as plain text. Goodness knows there are enough formats and structural editors out there, so in principle we only need to standardize on one, but it seems none of them are actually quite good enough yet.
    - jacques_chester 6y ago
      I am generally of the view that Avro is Good Enough for most things that plaintext is used for and is pretty well-supported. Arrow looks very promising for cases where fast raw data shipment is the goal.
      - andrewflnr 6y ago
        The tooling, though, we need the ubiquitous tooling. But that's not really a technical problem. :P Maybe when I pitch my hat in the structural editor ring I'll try to do an Avro editor.
jandrese 6y ago
Binary protocols are their own negative externality as well. They can be much harder to debug, requiring specialized tooling that may also have its own bugs. They can also suffer from things like insufficiently specified byte ordering issues and differences in floating point behaviour between systems. I know of at least one game that very close but not quite cross compatible between Windows and Linux due to differences in the way the floating point numbers are handled. People think they can just sweep all of that parsing complexity under the rug by just reading the bytes into their memory structures, but it comes back to bite them in the end.
- jacques_chester 6y ago
  > requiring specialized tooling that may also have its own bugs The common ones all have Wireshark plugins. So I'm not sure what's special. > They can also suffer from things like insufficiently specified byte ordering issues and differences in floating point behaviour between systems ... I know of at least one game that very close but not quite cross compatible between Windows and Linux... I think this shows I did a poor job of explaining myself. I don't mean that everyone should create their own binary encoding. I'm saying that you should pick a well-known, well-supported encoding like protobufs, Avro, CBOR, flatbuffers ... There are about a dozen strong contenders, all of which have tooling, library support and the ability to ROFLstomp plaintext on any measure of burden other than yours or mine.
- account42 6y ago
  > I know of at least one game that very close but not quite cross compatible between Windows and Linux due to differences in the way the floating point numbers are handled. People think they can just sweep all of that parsing complexity under the rug by just reading the bytes into their memory structures, but it comes back to bite them in the end. There is no difference in the binary encoding of floating point numbers between Linux and Windows - any differences there are (default rounding mode, etc) affect calculations and would be the same with a text-based format. In fact, text-based formats make things worse because people tend to forget that C parsing functions take the locale into account which affects the symbol used for the decimal point in floating point numbers.

andrewflnr 6y ago

Interesting perspective, but considering how much everyone seems to want software, I think we have to say that the commons also captures a lot of the same value from plain text that the programmers do. That might work out to less of a negative externality than just a trade-off for everyone, especially when you consider the positive effects on the commons of the network effects that text makes easier.

jacques_chester 6y ago
Interesting counterpoint. It would come down to how things net out. And it won't be stable over time. I'm still of the view that programmers systematically overvalue their convenience, because it's a value/cost tradeoff that they directly experience.
- andrewflnr 6y ago
  Fair. I do think the long term goal should be a compact binary format with equal or better tooling as plain text. Goodness knows there are enough formats and structural editors out there, so in principle we only need to standardize on one, but it seems none of them are actually quite good enough yet.
  - jacques_chester 6y ago
    I am generally of the view that Avro is Good Enough for most things that plaintext is used for and is pretty well-supported. Arrow looks very promising for cases where fast raw data shipment is the goal.
    - andrewflnr 6y ago
      The tooling, though, we need the ubiquitous tooling. But that's not really a technical problem. :P Maybe when I pitch my hat in the structural editor ring I'll try to do an Avro editor.

jandrese 6y ago

Binary protocols are their own negative externality as well. They can be much harder to debug, requiring specialized tooling that may also have its own bugs. They can also suffer from things like insufficiently specified byte ordering issues and differences in floating point behaviour between systems. I know of at least one game that very close but not quite cross compatible between Windows and Linux due to differences in the way the floating point numbers are handled. People think they can just sweep all of that parsing complexity under the rug by just reading the bytes into their memory structures, but it comes back to bite them in the end.

jacques_chester 6y ago
> requiring specialized tooling that may also have its own bugs The common ones all have Wireshark plugins. So I'm not sure what's special. > They can also suffer from things like insufficiently specified byte ordering issues and differences in floating point behaviour between systems ... I know of at least one game that very close but not quite cross compatible between Windows and Linux... I think this shows I did a poor job of explaining myself. I don't mean that everyone should create their own binary encoding. I'm saying that you should pick a well-known, well-supported encoding like protobufs, Avro, CBOR, flatbuffers ... There are about a dozen strong contenders, all of which have tooling, library support and the ability to ROFLstomp plaintext on any measure of burden other than yours or mine.
account42 6y ago
> I know of at least one game that very close but not quite cross compatible between Windows and Linux due to differences in the way the floating point numbers are handled. People think they can just sweep all of that parsing complexity under the rug by just reading the bytes into their memory structures, but it comes back to bite them in the end. There is no difference in the binary encoding of floating point numbers between Linux and Windows - any differences there are (default rounding mode, etc) affect calculations and would be the same with a text-based format. In fact, text-based formats make things worse because people tend to forget that C parsing functions take the locale into account which affects the symbol used for the decimal point in floating point numbers.