5 ms·

I know JSON is the standard now, but are there “better” serialization formats out there? Especially since JSON doesn’t know what an integer is in the spec

Analemma_ 1y ago
I'm having a vision of XML reading your comment and going "well well well, look who's decided to come crawling back".
- zahlman 1y ago
  I can't imagine why. XML is still fundamentally, well, a markup language, not a serialization format designed as such. But the "extensible" part isn't so accurate - attributes aren't extensible. GP complains that JSON doesn't know what an integer is (as distinct from a generic number), but at least it does know more than just strings. And needing to repeat a tag name when closing it just adds useless complexity.
  - wpm 1y ago
    It’s not anymore useless than a closing } or ], except since it has the tag name in it, so when I’m reading a highly nested object I’m not stuck in my text editor looking at a bunch of }’s at random indentation levels I have to scroll all the way back up to regain any context for. Tags are text which is visual structure I can choose to read, or choose to gloss over and use as bulk to shape the data in my head.
deleted 1y ago
[deleted]
bluGill 1y ago
Define better. As the other poster said, you could use XML which is more powerful, but as a result is a lot more complex. For most tasks I'd prefer JSON because while it is lacking, all the real world parsers I've seen are much easier to work with and I rarely need more complexity. If someone did a JSON++ (I have no doubt many people have but I'm not aware of them!) that added things like integers, without the complexity of XML that might be even better. In the real world if something should be an integer it isn't hard to check that and error out - you need to support parse errors in any data format anyway. Protobuf is sometimes better for data serialization. It isn't human readable, but you rarely need that and saving data bytes is often useful even today. Protobuf does have your integer type that you are missing, but it has other limitations might or might not apply to you. (I don't use protobuf enough myself to know what they are. Sqlite has more than once suggested that their database file is a great serialization format. You get a lot of power here and for complex things a database is often easier to work with than an xml file. There are various no sql databases as well that sometimes can work for this. I've handwritten my own serialization format in the past. The only hard part is designing enough the ability to add whatever the future needs are (note that I've never had to read my serialization on a different CPU family, things like little vs big endian I'm told can be a pain) There might be something else I didn't cover... Everything has pros and cons.
- bglusman 1y ago
  Protobuf does support JSON encoding[0], which I like as the .proto definition is quite readable, and then you can encode/decode either human readably or efficiently. It's even quite easy to have your consumer support both since the two are pretty easy to tell apart and if you know its either one or the other, you can just failover trying one to the other, possibly at some small cost... the guide also does point out some significant downsides to relying on the JSON version, but it can be useful in development and/or debugging in some cases, especially if you control both sides sending and receiving and can just toggle to use it when you want temporarily. [0]https://protobuf.dev/programming-guides/json/ https://protobuf.dev/programming-guides/json/
- ackfoobar 1y ago
  > It isn't human readable This is a tooling problem. Wireshark can decode protobuf for you when you're inspecting gRPC traffic.
  - horsawlarway 1y ago
    Needing that tooling is a format problem. JSON is bad at everything except being simple and easy. Turns out simple and easy is a real winner.
    - nothrabannosir 1y ago
      JSON has one glaring flaw: nested json encoding in strings becomes awful to read. I encounter it too often in reality where individual layers use JSON, but want to support arbitrary strings in their API. Encodings which use prefix length don't suffer from this, which ironically even includes most binary formats.
      - bluGill 1y ago
        Back to my main point though: normally I don't need the complexity that things like nested JSON would be. When you do though JSon is a bad format. (actually I would go so far as to say you never need something that complex - but the problems you are trying to solve with nested JSON are still complex enough that you should use a more powerful/complex framework, but better design of your data store would avoid the need for nested JSON.)
bqmjjx0kac 1y ago
There's CBOR, but it is not nearly as compact as the C in its name implies.
horsawlarway 1y ago
I guess it depends on how you define "better". JSON does a couple things really well, and most other things terribly. But the things it does well are pretty valuable. So in the "strengths" category I'd put the two following points: 1. JSON is very easy to read and understand as a human 2. JSON stuck to the basics. No comments, no references, no clever tricks, and not much space to let folks try to hammer in cleverness (see - no comments). Neither of those are all that much related to JSON itself as a format - the semantics are basically an accident of timing around JS syntax from the 2000s. But it's very, very useful to be able to get the raw text for a network message and know exactly what's getting sent without having to have a whole specialized tool framework to parse and understand the message. It's also useful to not let the spec get so complex that I never want to do that, even if I could (see: xml). So with JSON - I can easily read the actual network request and understand it, even with essentially zero additional tooling AND I have a very good chance of literally being able to open a text editor and create a new message with valid syntax without any other tools or references. Further - this holds true even if I'm not an industry expert with 20 years of experience. Most random people off the street can do it with only a couple minutes of coaching. Not many other serialization formats can do that. Imagine taking your 8 year old, sitting them down in front of the computer and legitimately saying "JSON doesn't know what an integer is in the spec"! It's true... but it's absolutely not the point. For normal people "number" is complex enough. And if you need an int and not a float... you can do that processing just fine after getting a JSON payload if you'd like. It won't be as fast as a specialized format (ex protobuffs), or as flexible (ex XML) as other formats - but that's a far distant concern to "Can I hold the hammer". JSON is really easy. "Easy" as a strength is wildly discounted, but man is it a winner when you get it. I also think it's surprisingly hard to do.
- RHSeeger 1y ago
  > No comments There are a large number of people that consider that _not_ a benefit.
  - knome 1y ago
    sure, but you'd see folks using them to add metadata and extend json in horrible incompatible ways
frankfrank13 1y ago
If you care a lot you can use Protobufs. Downside is now everything has to speak protobuf + no can't read in your network tab. Upside is (mostly) smaller payloads and a lot more type safety.
wpm 1y ago
It does, and it predates JSON https://www.apple.com/DTDs/PropertyList-1.0.dtd https://www.apple.com/DTDs/PropertyList-1.0.dtd As elegant as anything on json.org
zzo38computer 1y ago
Different formats are good for different things, but I think DER is much better. No character escaping is necessary, Unicode is not required (although it can be used if you want to do), arbitrary binary data can be stored, integers can be arbitrarily big (although implementations might only support integers as big as the specific application requires), you can skip past any block without needing to know how to interpret it, and many other advantages. (However, I had made up a variant with a few additional types, such as: key/value list, BCD string, TRON string, etc. This makes it strictly a superset of the types of data which can be stored in JSON (if the types you use are: sequence, key/value list, real number, null, boolean, and UTF-8 string). I use DER in some of my programs, because I think it is generally much better than JSON. Also, DER is a binary format, although I did make up a text format (called TER) which can be converted to DER (but TER is not really meant for other uses, since it is more complicated to handle).)

Analemma_ 1y ago

I'm having a vision of XML reading your comment and going "well well well, look who's decided to come crawling back".

zahlman 1y ago
I can't imagine why. XML is still fundamentally, well, a markup language, not a serialization format designed as such. But the "extensible" part isn't so accurate - attributes aren't extensible. GP complains that JSON doesn't know what an integer is (as distinct from a generic number), but at least it does know more than just strings. And needing to repeat a tag name when closing it just adds useless complexity.
- wpm 1y ago
  It’s not anymore useless than a closing } or ], except since it has the tag name in it, so when I’m reading a highly nested object I’m not stuck in my text editor looking at a bunch of }’s at random indentation levels I have to scroll all the way back up to regain any context for. Tags are text which is visual structure I can choose to read, or choose to gloss over and use as bulk to shape the data in my head.

deleted 1y ago

[deleted]

bluGill 1y ago

Define better. As the other poster said, you could use XML which is more powerful, but as a result is a lot more complex. For most tasks I'd prefer JSON because while it is lacking, all the real world parsers I've seen are much easier to work with and I rarely need more complexity. If someone did a JSON++ (I have no doubt many people have but I'm not aware of them!) that added things like integers, without the complexity of XML that might be even better. In the real world if something should be an integer it isn't hard to check that and error out - you need to support parse errors in any data format anyway. Protobuf is sometimes better for data serialization. It isn't human readable, but you rarely need that and saving data bytes is often useful even today. Protobuf does have your integer type that you are missing, but it has other limitations might or might not apply to you. (I don't use protobuf enough myself to know what they are. Sqlite has more than once suggested that their database file is a great serialization format. You get a lot of power here and for complex things a database is often easier to work with than an xml file. There are various no sql databases as well that sometimes can work for this. I've handwritten my own serialization format in the past. The only hard part is designing enough the ability to add whatever the future needs are (note that I've never had to read my serialization on a different CPU family, things like little vs big endian I'm told can be a pain) There might be something else I didn't cover... Everything has pros and cons.

bglusman 1y ago
Protobuf does support JSON encoding[0], which I like as the .proto definition is quite readable, and then you can encode/decode either human readably or efficiently. It's even quite easy to have your consumer support both since the two are pretty easy to tell apart and if you know its either one or the other, you can just failover trying one to the other, possibly at some small cost... the guide also does point out some significant downsides to relying on the JSON version, but it can be useful in development and/or debugging in some cases, especially if you control both sides sending and receiving and can just toggle to use it when you want temporarily. [0]https://protobuf.dev/programming-guides/json/ https://protobuf.dev/programming-guides/json/
ackfoobar 1y ago
> It isn't human readable This is a tooling problem. Wireshark can decode protobuf for you when you're inspecting gRPC traffic.
- horsawlarway 1y ago
  Needing that tooling is a format problem. JSON is bad at everything except being simple and easy. Turns out simple and easy is a real winner.
  - nothrabannosir 1y ago
    JSON has one glaring flaw: nested json encoding in strings becomes awful to read. I encounter it too often in reality where individual layers use JSON, but want to support arbitrary strings in their API. Encodings which use prefix length don't suffer from this, which ironically even includes most binary formats.
    - bluGill 1y ago
      Back to my main point though: normally I don't need the complexity that things like nested JSON would be. When you do though JSon is a bad format. (actually I would go so far as to say you never need something that complex - but the problems you are trying to solve with nested JSON are still complex enough that you should use a more powerful/complex framework, but better design of your data store would avoid the need for nested JSON.)

bqmjjx0kac 1y ago

There's CBOR, but it is not nearly as compact as the C in its name implies.

horsawlarway 1y ago

I guess it depends on how you define "better". JSON does a couple things really well, and most other things terribly. But the things it does well are pretty valuable. So in the "strengths" category I'd put the two following points: 1. JSON is very easy to read and understand as a human 2. JSON stuck to the basics. No comments, no references, no clever tricks, and not much space to let folks try to hammer in cleverness (see - no comments). Neither of those are all that much related to JSON itself as a format - the semantics are basically an accident of timing around JS syntax from the 2000s. But it's very, very useful to be able to get the raw text for a network message and know exactly what's getting sent without having to have a whole specialized tool framework to parse and understand the message. It's also useful to not let the spec get so complex that I never want to do that, even if I could (see: xml). So with JSON - I can easily read the actual network request and understand it, even with essentially zero additional tooling AND I have a very good chance of literally being able to open a text editor and create a new message with valid syntax without any other tools or references. Further - this holds true even if I'm not an industry expert with 20 years of experience. Most random people off the street can do it with only a couple minutes of coaching. Not many other serialization formats can do that. Imagine taking your 8 year old, sitting them down in front of the computer and legitimately saying "JSON doesn't know what an integer is in the spec"! It's true... but it's absolutely not the point. For normal people "number" is complex enough. And if you need an int and not a float... you can do that processing just fine after getting a JSON payload if you'd like. It won't be as fast as a specialized format (ex protobuffs), or as flexible (ex XML) as other formats - but that's a far distant concern to "Can I hold the hammer". JSON is really easy. "Easy" as a strength is wildly discounted, but man is it a winner when you get it. I also think it's surprisingly hard to do.

RHSeeger 1y ago
> No comments There are a large number of people that consider that _not_ a benefit.
- knome 1y ago
  sure, but you'd see folks using them to add metadata and extend json in horrible incompatible ways

frankfrank13 1y ago

If you care a lot you can use Protobufs. Downside is now everything has to speak protobuf + no can't read in your network tab. Upside is (mostly) smaller payloads and a lot more type safety.

wpm 1y ago

It does, and it predates JSON https://www.apple.com/DTDs/PropertyList-1.0.dtd https://www.apple.com/DTDs/PropertyList-1.0.dtd As elegant as anything on json.org

zzo38computer 1y ago

Different formats are good for different things, but I think DER is much better. No character escaping is necessary, Unicode is not required (although it can be used if you want to do), arbitrary binary data can be stored, integers can be arbitrarily big (although implementations might only support integers as big as the specific application requires), you can skip past any block without needing to know how to interpret it, and many other advantages. (However, I had made up a variant with a few additional types, such as: key/value list, BCD string, TRON string, etc. This makes it strictly a superset of the types of data which can be stored in JSON (if the types you use are: sequence, key/value list, real number, null, boolean, and UTF-8 string). I use DER in some of my programs, because I think it is generally much better than JSON. Also, DER is a binary format, although I did make up a text format (called TER) which can be converted to DER (but TER is not really meant for other uses, since it is more complicated to handle).)