5 ms·

Welcome to the world where everyone is wasting time and space on JSON encoding rather than using some sensible serialization protocol that can handle raw bytes.

mikenew 2y ago
Without any explicit admission of guilt on my part; what are some good options here? Protobuf is cool but I really don't want to mess with a special compiler and all that.
- LeoPanthera 2y ago
  Common alternatives are Apache Avro, MessagePack, FlatBuffers, and Thrift. I have no idea which is best. Frankly it seems like there are too many choices. Probably MessagePack, since it is JSON-like and I've actually heard of it.
  - ForHackernews 2y ago
    Avro has some really cool features like inbuilt schemas, schema versioning and migration (e.g. deprecating or renaming fields) but you pay for them with more overhead than MessagePack.
    - unscaled 2y ago
      Protocol Buffers have schemas too (though versioning and ensuring compatibility is quite messy and requires understanding internals). And it has less overhead than MesssagePack. I'm not sure what Avro is doing, but as a rule schema enables you to have less overhead, rather than more. The main advantage of MessagePack over schema-based formats is that it's dead-simple and mostly compatible with JSON. Schema-based formats usually need either a code generator or maintaining an annotated version of your data classes and making sure they match the schema. (Of course, with JSON or MessagePack you might still end up using a serialization library and something like JSON Schema).
      - ForHackernews 2y ago
        Oh, as I understand it Avro's schemas aren't just "built in" as in supported as a first-class part of the protocol, but rather that each message includes with it the schema needed to interpret it. This adds overhead to every message (it's still a binary protocol, though) but crucially it avoids a whole category of hassles around schema distribution and updating.
- chris_pie 2y ago
  CBOR could also be a good fit
  - crest 2y ago
    I just can't get over Cabo's choice to put 65 (yes 65) Bit fixed sized ints in the wire format and to top it off by making it ones complement (the sign bit is part of the type and the longest fixed sized ints are 64 bit range)...
    - recursive 2y ago
      Not familiar with this at all, but I assume it's so that one type can encode and store mixed lists of 64-bit signed and unsigned integers.
- DaiPlusPlus 2y ago
  HTTP multi-part. It’s a decades-solved problem and works with whatever the current request compression scheme is (gzip, brotli, etc).
  - sitharus 2y ago
    It’s not even an HTTP invention, it’s RFC2046 MIME from 1996. RFC2388 standardised its use in HTTP in 1998. The elegant thing about MIME is it allows multiple encodings and cross-references, so you can have your HTML and the images displayed in the HTML both optimally encoded in the same document, which was handy back in the time when HTML emails were taking off and marketing insisted that the fancy image signature they designed had to show every time, even when the person was reading the email offline… Of course back then we had to encode the image in base64 anyway because of non-8-bit-clean email servers. But I digress and will go back to my rocking chair.
- Spivak 2y ago
  The responses you got I think literally answer your question but probably aren't what you're going to reach for any kind of HTTP based thing. Your go-to should probably be multipart/form-data. It's well supported in every language and HTTP library and you can send both JSON and the file in the same payload. There seems to be a common trend of people writing "JSON APIs" thinking that every other part of HTTP is off-limits.
- odo1242 2y ago
  If you're looking for the simplest possible replacements, MessagePack (for data in transit) and SQLite (for data at rest) are worth considering
poincaredisk 2y ago
Wrong hill to die on IMO. I'm saving myself and others a lot of work by using a format that every modern language understands, without any external dependencies. It matters to me because I do a lot of integrations and if every service used their own bespoke format I would go mad. And the wire size is not that be - first, because most messages are small anyway (unless somebody is doing something stupid like sending files via json), and second, because HTTP compression is there and it works great for text formats like JSON.
- monocasa 2y ago
  > unless somebody is doing something stupid like sending files via json Like mp3s?
- mofoteam 2y ago
  [flagged]
- TeMPOraL 2y ago
  So, you're adding overhead of compression, decompression, parsing and serializing JSON at every step. All likely backed by a language where computing length of a string is O(n). And people are surprised software keeps being slow despite increase in compute resources. This is insane.
  - waterTanuki 2y ago
    There is an alternate universe where you comment how insane it is engineers waste time on trying to understand the nuances of every single bespoke byte encoding/decoding technique used between services. JSON is just fine for most tasks, otherwise the world would be in flames right now
    - Barrin92 2y ago
      I mean the world is kind of in flames, if planes were as unreliable as most software is we'd have a hundred falling out of the skies every day. But the only reason it isn't worse, and I can't remember who the quote is from, is that the real Moore's law is the fact that talented hardware engineers keep improving hardware more quickly than software is getting slower. There's an alternative world where everyone is performance oriented and those Rabbit devices don't run for five hours but for five days on modern batteries. Let's be real the thing is basically a Tamagochi, it should cost 50 bucks and run on chips from 2010.
foobiekr 2y ago
They will not understand your point. It is a kind of invincible ignorance, a little like how rich kids can’t understand why anyone has to budget. We’ve squandered almost all of the advancements.
Aleksdev 2y ago
I get what your saying but it’s what everyone uses. At this point it’s like saying write assembly instead of C. Hopefully people come around but I doubt it.

mikenew 2y ago

Without any explicit admission of guilt on my part; what are some good options here? Protobuf is cool but I really don't want to mess with a special compiler and all that.

LeoPanthera 2y ago
Common alternatives are Apache Avro, MessagePack, FlatBuffers, and Thrift. I have no idea which is best. Frankly it seems like there are too many choices. Probably MessagePack, since it is JSON-like and I've actually heard of it.
- ForHackernews 2y ago
  Avro has some really cool features like inbuilt schemas, schema versioning and migration (e.g. deprecating or renaming fields) but you pay for them with more overhead than MessagePack.
  - unscaled 2y ago
    Protocol Buffers have schemas too (though versioning and ensuring compatibility is quite messy and requires understanding internals). And it has less overhead than MesssagePack. I'm not sure what Avro is doing, but as a rule schema enables you to have less overhead, rather than more. The main advantage of MessagePack over schema-based formats is that it's dead-simple and mostly compatible with JSON. Schema-based formats usually need either a code generator or maintaining an annotated version of your data classes and making sure they match the schema. (Of course, with JSON or MessagePack you might still end up using a serialization library and something like JSON Schema).
    - ForHackernews 2y ago
      Oh, as I understand it Avro's schemas aren't just "built in" as in supported as a first-class part of the protocol, but rather that each message includes with it the schema needed to interpret it. This adds overhead to every message (it's still a binary protocol, though) but crucially it avoids a whole category of hassles around schema distribution and updating.
chris_pie 2y ago
CBOR could also be a good fit
- crest 2y ago
  I just can't get over Cabo's choice to put 65 (yes 65) Bit fixed sized ints in the wire format and to top it off by making it ones complement (the sign bit is part of the type and the longest fixed sized ints are 64 bit range)...
  - recursive 2y ago
    Not familiar with this at all, but I assume it's so that one type can encode and store mixed lists of 64-bit signed and unsigned integers.
DaiPlusPlus 2y ago
HTTP multi-part. It’s a decades-solved problem and works with whatever the current request compression scheme is (gzip, brotli, etc).
- sitharus 2y ago
  It’s not even an HTTP invention, it’s RFC2046 MIME from 1996. RFC2388 standardised its use in HTTP in 1998. The elegant thing about MIME is it allows multiple encodings and cross-references, so you can have your HTML and the images displayed in the HTML both optimally encoded in the same document, which was handy back in the time when HTML emails were taking off and marketing insisted that the fancy image signature they designed had to show every time, even when the person was reading the email offline… Of course back then we had to encode the image in base64 anyway because of non-8-bit-clean email servers. But I digress and will go back to my rocking chair.
Spivak 2y ago
The responses you got I think literally answer your question but probably aren't what you're going to reach for any kind of HTTP based thing. Your go-to should probably be multipart/form-data. It's well supported in every language and HTTP library and you can send both JSON and the file in the same payload. There seems to be a common trend of people writing "JSON APIs" thinking that every other part of HTTP is off-limits.
odo1242 2y ago
If you're looking for the simplest possible replacements, MessagePack (for data in transit) and SQLite (for data at rest) are worth considering

poincaredisk 2y ago

Wrong hill to die on IMO. I'm saving myself and others a lot of work by using a format that every modern language understands, without any external dependencies. It matters to me because I do a lot of integrations and if every service used their own bespoke format I would go mad. And the wire size is not that be - first, because most messages are small anyway (unless somebody is doing something stupid like sending files via json), and second, because HTTP compression is there and it works great for text formats like JSON.

monocasa 2y ago
> unless somebody is doing something stupid like sending files via json Like mp3s?
mofoteam 2y ago
[flagged]
TeMPOraL 2y ago
So, you're adding overhead of compression, decompression, parsing and serializing JSON at every step. All likely backed by a language where computing length of a string is O(n). And people are surprised software keeps being slow despite increase in compute resources. This is insane.
- waterTanuki 2y ago
  There is an alternate universe where you comment how insane it is engineers waste time on trying to understand the nuances of every single bespoke byte encoding/decoding technique used between services. JSON is just fine for most tasks, otherwise the world would be in flames right now
  - Barrin92 2y ago
    I mean the world is kind of in flames, if planes were as unreliable as most software is we'd have a hundred falling out of the skies every day. But the only reason it isn't worse, and I can't remember who the quote is from, is that the real Moore's law is the fact that talented hardware engineers keep improving hardware more quickly than software is getting slower. There's an alternative world where everyone is performance oriented and those Rabbit devices don't run for five hours but for five days on modern batteries. Let's be real the thing is basically a Tamagochi, it should cost 50 bucks and run on chips from 2010.

foobiekr 2y ago

They will not understand your point. It is a kind of invincible ignorance, a little like how rich kids can’t understand why anyone has to budget. We’ve squandered almost all of the advancements.

Aleksdev 2y ago

I get what your saying but it’s what everyone uses. At this point it’s like saying write assembly instead of C. Hopefully people come around but I doubt it.