8 ms·

JSON loses its appeal once you're not targeting browsers and/or anything else that uses Javascript. Sure, it's simple and familiar, but it has serious drawbacks

• Anderkent 9y ago
  CBOR is pretty good if you just want to move data around.
  • pagnol 9y ago
    I didn't know CBOR, but it looks similar to BSON and MessagePack, or is there a significant difference?
    • heavenlyhash 9y ago
      Roughly that it isn't shit. I say that a bit tongue in cheek, but really: BSON has a monstrous amount of basic data types, some of which make very little sense and are very difficult to round-trip through other formats; and MessagePack had crippling basic sanity issues with what-are-bytes-vs-strings in its early history and that's something you simply can't recover from without declaring game over and making a new spec. CBOR has neither of these issues. It's almost completely isomorphic to json, plus supports byte arrays. It's very nice.
      • roller 9y ago
        In the feature category, I'd also mention direct support for unbounded collections. CBOR can be its own container format for streaming data.
      • zbentley 9y ago
        Does CBOR support lazy/partial deserialization? It's not a make-or-break feature, but the ability to pluck out individual fields from large MessagePack documents without deserializing the whole thing has been a significant performance/memory savings for me many times. I haven't played with flatbuffers, but it probably supports this as well, given its design. Are there other binary protocols that have first-class (i.e. present with a good API in the client libraries for all major languages) support for lazy/partial deserialization?
        • js8 9y ago
          CBOR is very similar to MsgPack, so yes (if you design the schema well enough).
      • ci5er 9y ago
        Is/Can CBOR be canonicalized (like ASN.1, for example) for the purpose of unambiguous generation and interpretation of hashes? EDIT: I am happy to see that it handles date-time. Not really my favorite part of the non-spec of JSON on that front...
        • heavenlyhash 9y ago
          You might be interested in IPLD in particular: https://ipld.io/ https://ipld.io/ They use CBOR as the underlying serialization, and have a deep interest in canonicalization and applications of hashing.
      • RantyDave 9y ago
        More cbor ranting... In Python you can replace json.loads/dumps with cbor.loads/dumps with no further changes. Going back again may prove to be an issue when you discover just how much it is that json can't do.
• pjmlp 9y ago
  Υep, binary formats should be used when performance matters. If the issue is debugging, writing a dump tool is a one time effort.
  • oneeyedpigeon 9y ago
    When performance matters being the operative concept. Don’t optimise prematurely and throw out all the benefits of a textual format in favour of a binary one, unless you need to.
• ricardobeat 9y ago
  As usual, it depends. See https://chadaustin.me/wp-uploads/perf-msvc-haswell.png https://chadaustin.me/wp-uploads/perf-msvc-haswell.png
• perlgeek 9y ago
  Please correct me if I'm wrong, but I've always had the impression that protobufs introduce more coupling than JSON. Example: when you add a new element to the API response, the recipient needs to update their schema. Is that correct? If yes, how do you deal with that in a microservice architecture where you have little control over consumers of your API?
  • vkjv 9y ago
    Yes, this is correct, but, it's also exactly what you want in microservices. You are free to make forward compatible API changes without a deployment dance. E.g., you can add a field in the service and the caller won't see it until it is prepared to handle it properly.
  • weego 9y ago
    I don't understand the issue. The contract between producer and consumer should never be fragile around additional values existing.
    • zbentley 9y ago
      > The contract between producer and consumer should never be fragile around additional values existing. ... it should never be fragile where fragility might cost more than it gains you. The open/closed principle is the right thing to follow in most cases, but not all. For example, choking on unrecognized additional fields by default in development is often a big time/bug-saver, since it gives you an early "hint" that things might not be speaking the same dialect to each other. Similarly, if you have a message convention that is heavy on "modifiers" which control irreversible things, sometimes it's better to be fragile rather than count on orchestration systems to work perfectly 100% of the time. Suppose you have a system for buying bicycles with a required field of "bike model", and modifiers of "color", "wheel size", etc. Now let's say your servers all got patched to expect a new modifier of "frame size" . . . except for one server which didn't restart due to a deployment bug, and is still serving traffic. If a client sends that unexpected additional modifier to the "old" server, and it gets ignored, this could result in sending a production order for an expensive unit with the wrong specifications. If not caught, that cost could be compounded by sending the wrong unit to the customer and pissing them off. Now, this is clearly not a problem with the message format, but rather with the orchestration/release system. But bugs in orchestration layers are incredibly common, and baking a bit of fragility into the messaging layer at the right places can help mitigate those bugs' impact.
• ansgri 9y ago
  JSON, like XML, is supported by everything, including not only general purpose languages, but also by more specialized environments like MATLAB. Also its compatibility across implementations is very good, excluding support for comments and nonfinite numbers. Contrary to YAML, I mean; there's little reason to use XML except for legacy or markup purposes. JSON-schema implementations are also rather widely available. In performance-sensitive machine-to-machine applications, I agree, better to use someting compiled, but for general-purpose interoperation and archive data storage (where you cannot know when and with what tool you'll want to open it), I've yet to see a compelling alternative.
  • wirrbel 9y ago
    CBOR or MsgPack are binary JSON-like encodings. ProtoBuf depend on the schema definitions and encoder/decoder generation. Without the schema, you cannot read the format (theoretically), if you want your data to be read easily, JSON or CBOR or MsgPack are your best options.
    • ithkuil 9y ago
      Encoding/decoding of protobufs can be done dynamically if you know the schema.
      • wirrbel 9y ago
        knowing the schema also means knowing the right revision of the schema. If you store messages for a while, this already can be a problem.
        • ithkuil 9y ago
          schemaless is never really schemaless: the code that accesses your data is implicitly assigning it a schema, so you'll have the same problem. FWIW, protobuf schemas are designed to be friendly for back and forward compatibility; unknown fields can be ignored and "required fields are generally considered harmful", interesting read: https://github.com/google/protobuf/issues/2497 https://github.com/google/protobuf/issues/2497
• deleted 9y ago
  [deleted]
• user5994461 9y ago
  >>> ... and the lack of schema. The json schema is standardized and available in most languages and platforms. http://json-schema.org/ http://json-schema.org/
  • anilgulecha 9y ago
    It's one schema pattern, but definitely not standardized. The homepage you link to also does not claim standardization.
    • jack9 9y ago
      The json schema isn't a standard but is ubiquitous in California and in advertising (IAB standards). I dont know of anyone using another, or why you might. It's like saying ruby isn't the only ruby language because it isn't a formal standard. So what?
      • lobster_johnson 9y ago
        Ruby is perhaps not an ideal example, since it is in fact a formal standard (ISO/IEC 30170). Admittedly not updated beyond 1.8, but still.
  • nly 9y ago
    Ever tried writing a JSON schema? It's horrific. I'd rather use Protobufs or Flatbuffers for specification and then rely on their built-in JSON serialisations.
    • user5994461 9y ago
      I wrote quite a few. I wouldn't call that horrific at all.
    • zbentley 9y ago
      I don't think JSON schema is bad at all, for what it is. Remember that the more modern binary protocols like protobuf/etc. don't serve the same niche as JSON schema at all: JSON schema validates external documents; i.e. you take a schema and make sure a document complies with it. Protobuf/flatbuffers/etc. validate internal documents, as they are encoded or decoded. If that's all you need, then the schema definitions for those products will be much friendlier than JSON-schema, but that's comparing non-equivalent things. Another, minor quibble is that a lot of the schema-validation code in e.g. protobuf or Thrift is often generated by the framework, and the generated code can be quite hairy or confusing. There exist code generators for JSON schema as well, but writing a JSON schema by hand isn't comparable to, say, generating code from the Thrift IDL.
• onion2k 9y ago
  Sure, it's simple and familiar, but it has serious drawbacks such as the time and CPU resources it takes for parsing and the lack of schema. Before looking for something 'better' than JSON, build a basic prototype of your app and see if your application actually needs to use something else. The data in most applications is so small and will parse so easily that actually measuring the CPU utilisation will be quite hard. 99.99% of applications aren't moving data around at the sort of speed where the format really matters. Protocol buffers are very clever and really cool but using them where you don't need them is a huge waste of effort. As for the lack of a schema that's not true. There is a schema. What's in the file is defined somewhere. For applications that consume JSON that's in your code rather than the file format.
  • falcolas 9y ago
    Protobuffers have one major advantage over JSON: the schemas are versioned. When your requirements for the payload inevitably change, Protobuffers can handle that with a lot less muss and fuss than JSON can. Plus, the byte count over the wire is much smaller - something that matters even in today's world of gigabit ethernet.
    • ninjakeyboard 9y ago
      You have to compile the protocol buffers so there is overhead. Also the messages aren't human readable eg through kafka so you need to write something to look through the messages. JSON is faster if your dev team need to get from A to B in a hurry, but protocol buffers are a generally a better long term choice if you can put in the effort up front. Both have pros and cons.
    • pegasuscollins 9y ago
      > Protobuffers can handle that with a lot less muss and fuss than JSON can. Are you speaking from personal experience? FWIW it doesn't match mine.
    • skybrian 9y ago
      Not sure what you mean by "schemas are versioned". Protobufs have backward compatibility rules so you can add fields without changing versions. I don't think there's anything about explicit versioning in the spec?
• redshirt 9y ago
  I’d skip protobufs too. They’re horrible for performance. Many data center apps blow more cycles there than on compute. For actual applications where performance is necessary, binary “strict” formats are he only way to go. Protobufs are nice for ease of use, but they’re horrible otherwise.
  • swalsh 9y ago
    Maybe there's a middle ground between the 2.... like ASN.1
    • ci5er 9y ago
      You mean a self-describing compact schematized format that's been around for 3 decades? Please promote this as hard as you can. Then, us old guys getting aged-out of industry will be able to get jobs again.
      • teddyh 9y ago
        You’d need to repackage it in some form to make it palatable for modern audiences. Let me take a quick stab. Make “ASN.2” or something, update the fundamental data types to modern standards (think of what changed from NeXTSTEP to macOS X; they changed from TIFF to PNG, from PostScript to PDF, etc. but kept the overall architecture the same). Consolidate and simplify the many options (but do not eliminate options entirely). And call it… JASN or something, which sounds like both JSON and ASN, and make up some acronym where the “J” stand for “Joint” (like in JPEG, which stands for Joint Photographic Experts Group). And, maybe most importantly, make freely licensed tooling available for both converting to and from ASN.1 and JSON, but also jgrep, jless, jcat, etc. (similar to zgrep, zless and zcat, etc.) Make these packages available for all common operating systems, readily installable using that plaform’s usual mechanism.
• pegasuscollins 9y ago
  When comparing the two, keep in mind that JSON is a self-descriptive format while protobuf is not.
• zbentley 9y ago
  JSON has drawbacks, but I think you skipped one of its biggest benefits: it encodes in reasonably readable plain text. Readable plain-text protocols remove a constant overhead from all debugging/introspection tasks. If you use, say protobufs, you'll still have to convert back into strings to see what was in individual messages. This makes reading, for example, logfiles or wireshark captures with "grep" more difficult. Sure, there are tools that you can throw before "| grep" that convert to text, but that's one more piece of cognitive/debugging overhead you have to remember, and one more thing that new developers have to know to do if they're used to the standard unix/text-oriented way. That benefit applies even if your messages (or logs of messages) are stored/viewed in a database that speaks your binary protocol and provides tools to search/see the data as text. Whether or not you have something like that, it will always (always) be routinely necessary, or at least incredibly helpful to productivity, to "print out or search the textual content of messages sent/received by my code" when developing and testing apps that communicate over they network. A text protocol has a huge advantage in that department--or rather, it has a very small advantage that is present in a very large number of development/testing tasks. Also, readable plain text is key. XML dialects, or e.g. Thrift's JSON format are plain text, but are not terribly readable. This is subjective, to be sure, but makes a big difference in practice. There are great reasons to use binary protocols, but any comparison that leaves out the benefits of text content that is readable "for free" without using per-protocol tools is lacking.
  • ThrowawayR2 9y ago
    > Readable plain-text protocols remove a constant overhead from all debugging/introspection tasks. As others have pointed out, text-based serialization has significant overhead. To save a few hours of debugging, you're willing to spend gigabytes of bandwidth, an uncountable number of CPU cycles and memory, and all the associated electricity and hosting costs over the lifetime of your service? Penny wise and pound foolish, to say the least. Text-based serialization formats are just plain stupid for any type of service that handles over 100 requests per second. A good developer is not hampered in the slightest by the fact that their serialization format isn't text.
    • pjmlp 9y ago
• ktamura 9y ago
  >but it has serious drawbacks such as the time and CPU resources it takes for parsing and the lack of schema. Ditto. And if you still prefer the schema-free approach of JSON but with faster encoding and decoding support across all common programming languages, take a look at MessagePack. Unlike JSON, MessagePack is optimized for speed and encodes common data types efficiently. Unlike Protobuf/gRPC, it requires no centralized schema. MessagePack: https://msgpack.org/ https://msgpack.org/